Image processing apparatus and method, electronic device, medium

Abstract

The present disclosure provides an image processing device and method, an electronic device and a medium, and belongs to the technical field of computers. The image processing device comprises: a photosensitive module configured to generate a corresponding electrical signal according to an incident optical signal; a control module configured to control a processing mode of the electrical signal generated by the photosensitive module in a time division multiplexing manner; and a processing module configured to process the electrical signal according to the processing mode determined by the control module to obtain a processing result, wherein the processing mode comprises a first processing mode based on frame vision and a second processing mode based on dynamic vision. According to the embodiment of the present disclosure, the first processing mode based on frame vision and the second processing mode based on dynamic vision can be fused, and an output image based on frames and / or an output image based on dynamic vision can be obtained.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image processing apparatus and method, electronic device, and medium. Background Technology

[0002] Traditional visual image acquisition methods typically rely on fixed-frequency frame acquisition, resulting in drawbacks such as high redundancy, high latency, high noise, low dynamic range, and high data volume. In image acquisition methods based on Dynamic Vision Sensors (DVS), only the addresses and information of pixels with changing light intensity are output, rather than passively reading out every pixel within a frame sequentially. This eliminates redundant data at the source and reduces power consumption. Related technologies usually employ either of the above methods for image acquisition, failing to effectively integrate the two acquisition approaches. Summary of the Invention

[0003] This disclosure provides an image processing apparatus and method, an electronic device, and a medium.

[0004] In a first aspect, this disclosure provides an image processing apparatus, comprising: a photosensitive module for generating a corresponding electrical signal based on an incident light signal; a control module for controlling a processing mode of the electrical signal generated by the photosensitive module in a time-division multiplexing manner; and a processing module for processing the electrical signal according to the processing mode determined by the control module to obtain a processing result; wherein the processing mode includes a first processing mode based on frame vision and a second processing mode based on dynamic vision.

[0005] Secondly, this disclosure provides an image processing method, which includes: generating a corresponding electrical signal based on an incident light signal; determining a processing mode for the electrical signal in the time dimension; processing the electrical signal using a time-division multiplexing method according to the processing mode to obtain a processing result; and obtaining an output image based on the processing result; wherein the processing mode includes a first processing mode based on frame vision and a second processing mode based on dynamic vision.

[0006] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the image processing method described above.

[0007] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor / processing core, implements the above-described image processing method.

[0008] The embodiments provided in this disclosure fuse a first processing mode based on frame vision and a second processing mode based on dynamic vision in a time-division multiplexing manner, so that the image processing device supports both the first processing mode based on frame vision and the second processing mode based on dynamic vision, thereby allowing the selection of frame-based output images and / or dynamic vision-based output images as needed.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0011] Figure 1 A block diagram of an image processing apparatus provided in an embodiment of this disclosure;

[0012] Figure 2 A schematic diagram of an image processing apparatus provided in an embodiment of this disclosure;

[0013] Figure 3 A schematic diagram of an image processing apparatus provided in an embodiment of this disclosure;

[0014] Figure 4 This is a schematic diagram illustrating the working process of an image processing apparatus provided in an embodiment of the present disclosure;

[0015] Figure 5 This is a schematic diagram illustrating the working process of an image processing apparatus provided in an embodiment of the present disclosure;

[0016] Figure 6 This is a schematic diagram illustrating the connection between a comparison unit and a photosensitive sensor, provided in an embodiment of this disclosure.

[0017] Figure 7 This is a schematic diagram illustrating the connection between a comparison unit and a photosensitive sensor, provided in an embodiment of this disclosure.

[0018] Figure 8 This is a schematic diagram illustrating the connection between a comparison unit and a photosensitive sensor, provided in an embodiment of this disclosure.

[0019] Figure 9 This is a schematic diagram illustrating the connection between a comparison unit and a photosensitive sensor, provided in an embodiment of this disclosure.

[0020] Figure 10 This is a schematic diagram illustrating the connection between a comparison unit and a photosensitive sensor, provided in an embodiment of this disclosure.

[0021] Figure 11 A schematic diagram of the distribution of a photosensitive sensor provided in an embodiment of this disclosure;

[0022] Figure 12 This is a schematic diagram illustrating the connection between a comparison unit and a photosensitive sensor, provided in an embodiment of this disclosure.

[0023] Figure 13 This is a schematic diagram illustrating the connection between a comparison unit and a photosensitive sensor, provided in an embodiment of this disclosure.

[0024] Figure 14 A flowchart of an image processing method provided in an embodiment of this disclosure;

[0025] Figure 15 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0028] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0030] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0031] Vision is one of the most powerful forms of human perception, allowing us to acquire information about our surroundings without physical contact. With the development of technologies such as computers, vision can be applied to machines. This method of utilizing the processing power of machines for visual processing is called machine vision. In other words, machine vision endows machines with visual perception, giving them scene perception capabilities similar to those of biological visual systems.

[0032] In related technologies, machine vision primarily relies on traditional cameras. Traditional cameras typically use frame-based image sensors to acquire images. When capturing images, they are usually based on "frames" acquired at a fixed frequency, with the image sensor obtaining the pixel values ​​of each pixel in each frame. While images acquired by traditional cameras are of high quality, the large amount of data processing, including capture, communication, and processing for each frame, can lead to significant redundancy and unnecessary data. This high data load often necessitates reducing temporal resolution to slow response time, resulting in increased power consumption and increasing the size and cost of the machine vision system. Furthermore, frame-based image sensors suffer from limited dynamic range, poor low-light performance, and motion blur because data is captured as a sequence of still images (frames). In other words, traditional cameras are suitable for applications requiring high image display quality (e.g., film and television production) or machine vision tasks such as object recognition. However, for machine vision tasks like tracking, monitoring, and motion estimation, frame-based traditional cameras no longer offer processing advantages.

[0033] Dynamic Vision Sensors (DVS) employ a novel image processing method based on Address-Event Representation (AER), mimicking the working mechanism of biological vision. Instead of passively reading out every pixel within a "frame," DVS only outputs the addresses and information of pixels where light intensity changes, eliminating redundant data at the source. It features real-time dynamic response to scene changes, ultra-sparse image representation, and asynchronous event output, allowing data processing with fewer resources, lower power consumption, and faster system response time. Consequently, the image quality acquired by DVS may be lower than that of frame-based image sensors. Based on these characteristics, DVS is widely used in machine vision tasks such as high-speed target tracking, real-time monitoring, and industrial automation.

[0034] In related technologies, the aforementioned frame-based image sensors and dynamic vision sensors are often used independently, without effective integration.

[0035] The image processing apparatus according to the embodiments of the present disclosure can fuse a first processing mode based on frame vision and a second processing mode based on dynamic vision in a time-division multiplexing manner, so that the image processing apparatus supports both the first processing mode based on frame vision and the second processing mode based on dynamic vision, thereby allowing the selection of frame-based output images and / or dynamic vision-based output images as needed.

[0036] The image processing method according to embodiments of this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server. In other words, the image processing apparatus according to embodiments of this disclosure can be packaged as an electronic device product such as a terminal device or a server.

[0037] In a first aspect, embodiments of this disclosure provide an image processing apparatus.

[0038] Figure 1 This is a block diagram of an image processing apparatus provided according to an embodiment of the present disclosure. (Refer to...) Figure 1 The image processing apparatus includes:

[0039] The photosensitive module 10 is used to generate a corresponding electrical signal based on the incident light signal.

[0040] The control module 20 is used to control the processing mode of the electrical signal generated by the photosensitive module in a time-division multiplexing manner, wherein the processing mode includes a first processing mode based on frame vision and a second processing mode based on dynamic vision.

[0041] The processing module 30 is used to process the electrical signal according to the processing mode determined by the control module and obtain the processing result.

[0042] In some possible implementations, the photosensitive module has a photosensitive function, which can convert light signals into corresponding electrical signals for further processing. The light signals include those reflected from the object to be imaged onto the photosensitive module.

[0043] In some possible implementations, the photosensitive module achieves its light-sensing function through a sensor.

[0044] In some possible implementations, the photosensitive module includes multiple photosensitive sensors, which generate corresponding electrical signals based on the intensity of the incident light signal.

[0045] In some possible implementations, the aforementioned photosensitive sensors can be arranged in an array to form a corresponding sensor array. Moreover, the smallest photosensitive unit of the photosensitive sensor corresponds to a pixel, and the photosensitive area of ​​the photosensitive sensor is composed of the arrayed pixels.

[0046] For incident light signals, a photosensor can detect their intensity and convert it into a matching electrical signal. Here, "matching" means that there is a certain mapping or proportional relationship between the light intensity and the intensity of the electrical signal, so that after processing the electrical signal, accurate image information can be obtained.

[0047] In some possible implementations, the electrical signal includes at least one of capacitance, voltage, and current signals. Correspondingly, photosensitive sensors employ various sensing methods to output different types of electrical signals.

[0048] For example, the photosensitive sensor includes at least one of a first sensor, a second sensor, and a third sensor; wherein the first sensor is used to acquire the light intensity of the light signal and generate a corresponding capacitance signal based on the light intensity, the second sensor is used to acquire the light intensity of the light signal and generate a corresponding voltage signal based on the light intensity, and the third sensor is used to acquire the light intensity of the light signal and generate a corresponding current signal based on the light intensity.

[0049] For example, the first sensor includes a storage capacitor that can obtain a capacitance signal C related to light intensity through charging and discharging operations; the second sensor includes a photosensitive signal circuit that can generate a voltage signal V that depends on light intensity; and the third sensor includes devices such as a photodiode (PD) that can measure impact light and convert light intensity into a corresponding current signal I.

[0050] It should be noted that the above examples of photosensitive sensors are merely illustrative, and the embodiments disclosed herein do not limit the types of photosensitive sensors and their output signals.

[0051] It should also be noted that the pixel module can be composed of any one of the above-mentioned photosensitive sensors, or it can be composed of two or more of the above-mentioned photosensitive sensors. This disclosure does not limit this.

[0052] After the photosensitive module generates an electrical signal based on the optical signal, the electrical signal needs to be further processed to obtain the output image. In this embodiment, the image processing device supports both a first processing mode based on frame vision and a second processing mode based on dynamic vision. Therefore, the control operation of relevant functional modules is required to determine which processing mode to use for processing the electrical signal, thereby ensuring the orderly execution of the image processing task.

[0053] In some possible implementations, the processing mode of the electrical signal is controlled by a control module. The processing modes include a first processing mode based on frame vision and a second processing mode based on dynamic vision.

[0054] For example, the first processing mode based on frame vision can acquire and output images frame by frame at a fixed frequency, while the second processing mode based on dynamic vision only outputs dynamic visual event data when it is determined that the light intensity has changed.

[0055] In some possible implementations, the control module can control the processing mode based on a time control circuit, that is, "gating" a first processing mode or a second processing mode in the time dimension through the time control circuit, so as to process the electrical signal based on the selected processing mode.

[0056] In some possible implementations, the control module can control the processing mode by sending control signals to the processing module. That is, the control signals include information about the processing mode specified by the control module. The processing module responds to the control signals sent by the control module, thereby determining which processing mode to use to process the electrical signal.

[0057] It should be understood that, regardless of the control mode adopted, the processing module's processing of electrical signals is essentially a time-division multiplexing method, that is, using the first processing mode in some time periods and the second processing mode in other time periods. The time periods corresponding to the first and second processing modes can be regular time periods or randomly set irregular time periods; this disclosure does not impose any limitations on this.

[0058] In some optional implementations, the control module is used to control the processing mode of the electrical signals generated by the photosensitive module according to a time-division multiplexing method. Time-division multiplexing means that a first processing mode is used to process the electrical signals during a portion of the time period (or a portion of the moment), while a second processing mode is used to process the electrical signals during the remaining time period (or the remaining moment). In the time dimension, this manifests as the first and second processing modes being used alternately to process the electrical signals.

[0059] The processing mode for electrical signals during which time periods is used can be determined based on real-time indication signals or information, or it can be determined according to a preset time period. This disclosure does not limit this.

[0060] For example, if the current processing mode is the first processing mode, the control module switches the electrical signal processing mode from the first processing mode to the second processing mode upon receiving an externally sent indication signal. Furthermore, if it receives another externally sent indication signal, it switches the electrical signal processing mode back to the first processing mode. The indication signal can be sent by a preset management terminal, management server, or other device.

[0061] For example, for the electrical signals generated by the photosensitive module in the first and second cycles, the control module controls the processing module to alternately process the electrical signals using a first processing mode and a second processing mode. In other words, the control module controls the processing module to alternately process the electrical signals using the first processing mode and the second processing mode in the first and second cycles, respectively, based on time-division multiplexing. The first and second cycles can have the same duration or different durations.

[0062] In other words, the processing module adopts the first processing mode in each first cycle and the second processing mode in each second cycle, and the first cycle and the second cycle are alternately set in the time dimension.

[0063] It is important to emphasize that traditional frame-based cameras involve the concept of frame rate. This is because reading the image data after exposure takes time, and during this reading process, the photosensitive module is no longer performing exposure processing, resulting in the photosensitive module being idle during the waiting period and thus having low utilization. In contrast, in this embodiment, the photosensitive module is used based on time-division multiplexing. Specifically, during the exposure phase, the photosensitive module is used to perform exposure processing, and during the reading phase, it is used to acquire dynamic visual event data. This method can utilize the photosensitive module even during the waiting period, thereby improving its utilization rate. In other words, through time-division multiplexing, the photosensitive module is used as a different sensor at different times, and together with the control module and processing module, they jointly complete data processing to obtain the corresponding processing results (including frame-based output images and / or dynamic vision-based output images).

[0064] For example, assuming that in a traditional camera, the exposure processing of frame data takes 10 milliseconds (ms) and the reading processing takes 50 ms, the photosensitive module is idle during the 50 ms corresponding to the reading operation. In this embodiment of the present disclosure, to improve the utilization rate of the photosensitive module, a first processing mode is set to correspond to 10 ms and a second processing mode to correspond to 50 ms, so that frame data is obtained through 10 ms exposure, and the 50 ms time for reading frame data is used to both read and process the exposed frame data and reuse the photosensitive module to acquire dynamic visual event data, thereby making full use of the entire time of the photosensitive module and effectively improving the utilization rate of the photosensitive module.

[0065] For example, if the first cycle corresponds to 1ms and the second cycle corresponds to 2ms, taking the start time corresponding to the first cycle as an example, then the first processing mode is used to process the electrical signal in the first 1ms, the second processing mode is used in the second and third ms, the first processing mode is used in the fourth ms, the second processing mode is used in the fifth and sixth ms, and so on. Taking the start time corresponding to the second cycle as an example, then the second processing mode is used to process the electrical signal in the first and second ms, the first processing mode is used in the third ms, the second processing mode is used in the fourth and fifth ms, the first processing mode is used in the sixth ms, and so on.

[0066] It should be noted that the above description of the control module is merely an example, and the embodiments disclosed herein do not limit the control module's control method for the processing mode.

[0067] After the control module determines the processing mode, the processing module can then perform the corresponding data processing operations according to the determined processing mode.

[0068] In some possible implementations, the processing module processes the electrical signal according to the processing mode determined by the control module to obtain the processing result, so as to obtain the output image based on the processing result.

[0069] In some possible implementations, the processing module includes a storage unit and a comparison unit; wherein, the control module is used to control the photosensitive module to connect to the storage unit to obtain a first processing result when a first processing mode is determined to be used, and / or, the control module is used to control the photosensitive module to connect to the comparison unit to obtain a second processing result when a second processing mode is determined to be used.

[0070] When the photosensitive module is connected to the storage unit, the storage unit is used to perform exposure processing on the electrical signal output by the photosensitive module to obtain a first processing result, which includes at least a pixel integral value. When the photosensitive module is connected to the comparison unit, the comparison unit is used to determine a signal difference based on the electrical signal output by the photosensitive module and a reference signal value, and obtain a second processing result based on the comparison result of the signal difference and a preset threshold. The second processing result includes outputting dynamic visual event data or not outputting dynamic visual event data.

[0071] In some possible implementations, dynamic visual event data includes at least the address and change information of the pixels whose light intensity changes.

[0072] In some possible implementations, the control module controls the processing module to process the electrical signal using a time-division multiplexing method, alternating between a first processing mode and a second processing mode in the first and second cycles, respectively.

[0073] Correspondingly, in the first cycle, the control module controls the storage unit to connect with the photosensitive module to obtain frame data; in the second cycle, the control module controls the comparison unit to connect with the photosensitive module, and determines whether to output dynamic visual event data based on the comparison result of the comparison unit.

[0074] For example, during the first cycle, the storage unit performs exposure processing on the electrical signal output by the photosensitive module to obtain the corresponding frame data.

[0075] For example, during the second cycle, the comparison unit calculates the signal difference based on the electrical signal output by the photosensitive module and the reference signal value, and compares the signal difference with a preset threshold to obtain a comparison result. If the comparison result is that the signal difference is greater than the preset threshold, the unit determines to output dynamic visual event data; otherwise, it determines not to output dynamic visual event data.

[0076] In some possible implementations, the comparison unit operates based on a preset time step. Accordingly, at the i-th time step, the comparison unit determines the signal difference based on the electrical signal output by the photosensitive module at the i-th time step and the electrical signal at the (i-1)-th time step, and if the signal difference is greater than a preset threshold, determines the second processing result as outputting dynamic visual event data.

[0077] In some optional implementations, the reference signal value can be determined based on the electrical signal output by the photosensitive module at the previous time step in the current time step. It should be noted that the reference signal value can also be set in other ways (for example, determining the electrical signal at a certain time before the current time as the reference signal value), and the preset threshold can be set based on any one or more of the following: experience, statistical data, actual needs, etc. The embodiments of this disclosure do not limit the method of setting the reference signal value and the preset threshold.

[0078] As can be seen from the above, the reference signal value is a benchmark value used to determine whether the electrical signal has changed, and the preset threshold is used to measure whether the range of change of the electrical signal is sufficient to output the corresponding dynamic visual event data.

[0079] Taking a 10ms exposure processing and a 50ms read processing time for frame data as an example, the photosensitive module performs exposure processing within the 1-10ms timeframe. Within 11-50ms, it acquires dynamic visual event data and simultaneously reads the pixel integral values ​​acquired within the 1-10ms timeframe to obtain the frame image data corresponding to that timeframe. Within 51-60ms, the photosensitive module performs exposure processing again. Within 61-110ms, it again acquires dynamic visual event data and reads the pixel integral values ​​acquired within the 51-60ms timeframe to obtain the frame image data corresponding to that timeframe, and so on, thus obtaining image data for each timeframe.

[0080] As shown above, for time periods such as 1-10ms and 51-60ms, dynamic visual event data is missing because the photosensitive module is used for exposure processing. However, this method still has strong adaptability for scenarios with slow scene changes or slow object movement.

[0081] It should be noted that in the first processing mode, each frame of the image needs to be processed. Taking a frame rate of 30 as an example, 300 images can be generated in 10 seconds. Due to the need for capturing and communicating with each pixel, the processing volume is large, and the power consumption is correspondingly high. In contrast, in the second processing mode, if a target object only moves at a certain moment and remains in the same position at other times, dynamic visual event data will only be output at the moment of displacement. There will be no output at other times. Converted to a two-dimensional structure, this manifests as a certain degree of data sparsity, resulting in a smaller data processing volume and correspondingly lower power consumption.

[0082] In summary, the first processing mode can obtain relatively high-quality image frames, but it processes a large amount of data and loses dynamic visual event data within that time period. The second processing mode can obtain the corresponding dynamic visual event data and processes a smaller amount of data, but the image quality is relatively lower. Regarding the time allocation between the first and second processing modes (e.g., the length relationship between the first and second cycles), the time allocation ratio can be determined by static setting or dynamic adjustment according to the application scenario, processing volume requirements, etc., and this disclosure does not impose any limitations on this.

[0083] For example, a first processing mode is set to correspond to a first cycle, and a second processing mode is set to correspond to a second cycle. That is, the frame data exposure time is the first cycle, the dynamic visual event data acquisition time is the second cycle, and the exposure data obtained in the adjacent previous first cycle is also read and processed within the second cycle.

[0084] For example, while meeting the frame rate requirements, the working time of the second processing mode can be appropriately increased (e.g., the second cycle can be set to be longer than the first cycle) in order to obtain more dynamic visual event data.

[0085] For example, provided that dynamic visual event data that meets the task requirements is obtained, the working time of the first processing mode can be appropriately increased (e.g., the first cycle is set to be longer than the second cycle) to improve the image quality.

[0086] It should be noted that neither the first processing result obtained using the first processing mode nor the processing result corresponding to the second processing mode is yet in image form. Therefore, further processing by the corresponding functional units is required to obtain the output image. Moreover, since the first and second processing results are of different types, the corresponding processing methods may also be different. Therefore, separate functional units can be set for the first processing result (or the first processing mode) and the second processing result (or the second processing mode) to obtain their respective output images.

[0087] In some possible implementations, the processing module further includes a pixel reading unit connected to the storage unit. This pixel reading unit is used to read and process the data stored in the storage unit. In other words, the pixel reading unit is the functional unit corresponding to the first processing result, and can obtain the output image based on the first processing result.

[0088] In some possible implementations, the pixel reading unit can perform reading processing when the comparison unit in the processing module is in the second processing mode, so as to obtain the output image without affecting the image acquisition operation of the second processing mode, thereby improving processing efficiency.

[0089] For example, the data stored in the storage unit is the pixel integral value. The pixel reading unit reads and processes the pixel integral value of the storage unit in the previous first cycle adjacent to the second cycle in the second cycle to obtain the output image (i.e., frame data). The first cycle corresponds to the first processing mode, and the second cycle corresponds to the second processing mode.

[0090] In some possible implementations, the processing module further includes an encoding unit connected to the comparison unit. The encoding unit encodes the dynamic visual event data output by the comparison unit to obtain a pulse code, and acquires the output image based on the pulse code. In other words, the encoding unit is the functional unit corresponding to the second processing result, used to acquire the output image based on the second processing result.

[0091] For example, since the comparison unit only outputs dynamic visual event data for pixels with large changes, the dynamic visual event data output by the comparison unit can be represented as a pulse sequence. The encoding unit performs pulse encoding on the dynamic visual event data to obtain the pulse code, and further obtains the output image based on the pulse code.

[0092] It should be noted that the encoding process of the encoding unit can be completed within the second cycle or within the next first cycle adjacent to the second cycle. This disclosure does not limit this.

[0093] The following is combined Figures 2-5 The image processing apparatus according to embodiments of this disclosure will be described in detail.

[0094] Figure 2 This is a schematic diagram of an image processing apparatus provided according to an embodiment of the present disclosure. (Refer to...) Figure 2The image processing device includes a photosensitive module, a timing control circuit (equivalent to a control module), and a processing module. The processing module includes a storage unit, a pixel readout unit, a comparison unit, and an encoding unit. The timing control circuit includes three interfaces: interface T1, interface T2, and interface T3. Interface T1 is connected to the photosensitive module, interface T2 is connected to the storage unit of the processing module, and interface T3 is connected to the comparison unit of the processing module. The storage unit is connected to the pixel readout unit, and the comparison unit is connected to the encoding unit.

[0095] In some possible implementations, the photosensitive module generates a corresponding electrical signal based on the intensity of the incident light signal. The timing control circuit's interface T1 is connected to the photosensitive module, and the processing mode for the electrical signal is determined by controlling the connection between interface T1 and interface T2 or T3. Specifically, when interface T1 is connected to interface T2, the electrical signal is input to the storage unit. The storage unit performs exposure processing on the electrical signal to obtain the pixel integral value of each pixel and transmits the pixel integral value to the pixel readout unit. The pixel readout unit reads the pixel integral value of each pixel and arranges the pixel integral values ​​according to a corresponding array to obtain the output image. When interface T1 is connected to interface T3, the electrical signal is input to the comparison unit. The comparison unit determines the signal difference based on the electrical signal and a reference signal value, compares the signal difference with a preset threshold, and if the signal difference is less than or equal to the preset threshold, no dynamic visual event data is output. If the signal difference is greater than the preset threshold, the comparison unit outputs the dynamic visual event data and transmits the dynamic visual event data to the encoding unit. The encoding unit encodes the dynamic visual event data to obtain a pulse code, and the output image is obtained based on the pulse code.

[0096] As can be seen from the above, when interface T1 is connected to interface T2, the processing module performs image processing based on the first processing mode, which can obtain an output image with higher quality. When interface T1 is connected to interface T3, the processing module performs image processing based on the second processing mode, which can obtain an output image including dynamic visual event data.

[0097] It should be noted that, in addition to controlling the processing mode of electrical signals through a time control circuit, the processing mode of electrical signals can also be controlled through control signals. The following section will discuss this further. Figure 3 This describes an image processing device that uses a control signal to control the processing mode.

[0098] Figure 3 This is a schematic diagram of an image processing apparatus provided according to an embodiment of the present disclosure. (Refer to...) Figure 3The image processing device includes a photosensitive module, a control module, and a processing module. The processing module includes a storage unit, a pixel reading unit, a comparison unit, and an encoding unit. Both the photosensitive module and the control module are connected to the processing module. The photosensitive module transmits electrical signals to the processing module, and the control module sends control signals to the processing module.

[0099] In some possible implementations, the photosensitive module generates a corresponding electrical signal based on the intensity of the incident light signal and inputs the electrical signal into the processing module. The control module sends a control signal to the processing module, and after receiving the control signal, the processing module determines which processing mode to use to process the electrical signal. Specifically, when the control signal indicates that the processing module uses the first processing mode, the processing module performs exposure processing on the electrical signal based on the storage unit to obtain the pixel integral value of each pixel, and transmits the pixel integral value to the pixel readout unit. The pixel readout unit reads the pixel integral value of each pixel and arranges the pixel integral values ​​according to the corresponding array to obtain the output image. When the control signal indicates that the processing module uses the second processing mode, the processing module processes the electrical signal based on the comparison unit. The comparison unit determines the signal difference based on the electrical signal and the reference signal value, compares the signal difference with a preset threshold, and if the signal difference is less than or equal to the preset threshold, no dynamic visual event data is output. If the signal difference is greater than the preset threshold, the comparison unit outputs the dynamic visual event data and transmits the dynamic visual event data to the encoding unit. The encoding unit encodes the dynamic visual event data to obtain a pulse code, and obtains the output image based on the pulse code.

[0100] It should be noted that in this embodiment, the photosensitive module and the comparison unit are connected, and their combined operation enables the function of a dynamic vision sensor. In other words, the dynamic vision sensor only outputs the address and information of pixels whose light intensity has changed (i.e., dynamic visual event data). In this embodiment, the photosensitive module outputs an electrical signal that can characterize light intensity. The comparison unit is connected to the photosensitive module, thereby receiving the electrical signal output by the photosensitive module and determining whether the light intensity of the corresponding pixel has changed based on the electrical signal (achieved by calculating the signal difference between the electrical signal and a reference signal value, and comparing the signal difference with a preset threshold, etc.), and outputting dynamic visual event data when it is determined that the light intensity has changed. Through the above settings, the two processing modes can be fused simply by adding a comparison unit to a traditional image processing device, which is easy to operate. Furthermore, since the photosensitive module is shared, there is no need to add a dynamic vision sensor, thus effectively reducing the device cost.

[0101] Figure 2 and Figure 3 An image processing apparatus is exemplarily shown from a functional structural perspective. To further illustrate the image processing apparatus, based on... Figure 4 The operation of the image processing device is shown from a time perspective.

[0102] Figure 4 This is a schematic diagram illustrating the operation of an image processing apparatus provided in an embodiment of this disclosure. (Refer to...) Figure 4 It shows from the time dimension Figure 2 The image processing device shown illustrates the processing procedure.

[0103] like Figure 4 As shown, taking the time period t1-t2 as an example: During the t1-t2 time period, the photosensitive module is multiplexed to acquire dynamic visual event data. Simultaneously, the t1-t2 time period is also used to read and process the data obtained from the exposure during the t0-t1 time period to obtain the frame image (p1) corresponding to the t0-t1 time period. Similarly, the t2-t3 time period can also be multiplexed: During the t2-t3 time period, the photosensitive module is used for exposure processing to acquire exposure data. Simultaneously, the t2-t3 time period is also used to process the dynamic visual event data acquired during the t1-t2 time period to obtain the event image (p2) corresponding to the t1-t2 time period.

[0104] It should be noted that, Figure 4 In this context, p1-p5 represent the output images corresponding to each time period, but do not imply that the output image can be obtained within the corresponding time period. For example, p1 indicates that it is a frame image obtained based on the exposure data of the photosensitive module within the time period t0-t1, but it does not mean that p1 can be obtained within the time period t0-t1.

[0105] The following is based on Figure 4 The operation of the image processing apparatus in the embodiments of this disclosure will be described in detail.

[0106] like Figure 4As shown, during the time period t0-t1, interface T1 connects to interface T2. The processing module uses the first processing mode to process data, which is within the global pixel integration time period. It obtains the pixel integration value of each pixel. During the time period t1-t2, the pixel integration value of the time period t0-t1 is read through pixel reading and other operations to obtain the output image corresponding to the time period t1-t2 (the output image is shown in p1). At the same time, during the time period t1-t2, interface T1 connects to interface T3. The photosensitive module is multiplexed to convert the light signal into the corresponding electrical signal. The processing module uses the second processing mode to process the generated electrical signal to obtain dynamic visual event data of pixels whose signal difference exceeds a preset threshold. Based on the dynamic visual event data, an event image is obtained, and the corresponding output image is shown in p2. During the time interval t2-t3, interface T1 reconnects to interface T2, the photosensitive module performs exposure operations, and the processing module uses the first processing mode for data processing. During the global pixel integration time interval, it obtains the pixel integration values ​​of each pixel. During the time interval t3-t4, pixel reading and other operations are performed to read the pixel integration values ​​of the t2-t3 time interval, thereby obtaining the output image corresponding to the t2-t3 time interval (the output image is shown in p3). Similarly, during the time interval t3-t4, interface T1 reconnects to interface T3, and the photosensitive module is reused again to convert the light signal into the corresponding electrical signal. The processing module uses the second processing mode to process the generated electrical signal, obtaining dynamic visual event data of pixels whose signal difference exceeds a preset threshold. Based on this dynamic visual event data, an event image is obtained, thus obtaining the output image shown in p4. The time interval t4-t5 is similar to t0-t1 and t2-t3; for the t4-t5 time interval, the output image shown in p5 can be obtained. This process continues until the image processing task is completed.

[0107] In the above processing, only time periods such as t1-t2 and t3-t4 are reused. In some optional implementations, time periods such as t2-t3 and t4-t5 can also be reused.

[0108] For example, to obtain more dynamic visual event data, the lengths of time periods such as t1-t2 and t3-t4 are appropriately increased. Based on this setting, it may result in the pixel integral values ​​of the t0-t1 time period being read before time t2 arrives, and it may also result in the inability to obtain the output image corresponding to t1-t2 before or at time t2. Therefore, it is necessary to reuse the data in the t2-t3 time period to obtain the output image corresponding to t1-t2 based on the dynamic visual event data of t1-t2. The t4-t5 time period is similar and will not be described further here.

[0109] It should be noted that in some optional implementations, to improve image quality and obtain more frame images, the time corresponding to the second processing mode is matched with the frame data reading time. Taking the t1-t2 time period as an example, this means that the pixel integral value of the t0-t1 time period is read exactly at time t2, thereby obtaining the output image corresponding to the t1-t2 time period. In some optional implementations, the duration of the second processing mode is longer to obtain more dynamic visual event data. Taking the t1-t2 time period as an example, the pixel integral value of the t0-t1 time period can be read after time t1 but before time t2, thereby obtaining the output image corresponding to the t1-t2 time period.

[0110] In some possible implementations, the comparison unit operates based on a preset time step Δt. Taking the time interval t1-t2 as an example, starting at time t1, after the first time step, at time t1+Δt, the comparison unit uses the electrical signal at time t1 as the reference signal value for the current time step, calculates the difference between the electrical signal at time t1+Δt and the electrical signal at time t1, obtains the signal difference, and compares this signal difference with a preset threshold. When the signal difference is greater than the preset threshold, it indicates a significant change in the light intensity of the corresponding pixel. Therefore, the unit outputs dynamic visual event data corresponding to the first time step, including pixel position and difference information (used to characterize the change in light intensity). Conversely, if the signal difference is less than or equal to the preset threshold, it indicates no significant change in the light intensity of the corresponding pixel, and therefore, no dynamic visual event data is output. After another time step... Upon reaching time t1+2Δt, the comparison unit uses the electrical signal at time t1+Δt as the reference signal value for the current time step, calculates the difference between the electrical signal at time t1+2Δt and the electrical signal at time t1+Δt, obtains the signal difference value, and compares this signal difference value with a preset threshold. If the signal difference value is greater than the preset threshold, it indicates that the light intensity of the corresponding pixel has changed significantly, so dynamic visual event data corresponding to the second time step is output, including pixel position and difference information. Conversely, if the signal difference value is less than or equal to the preset threshold, it indicates that the light intensity of the corresponding pixel has not changed significantly, so dynamic visual event data is not output. This process continues until time t2, when the processing unit enters the first processing mode, and the comparison unit is in a paused or sleep state.

[0111] It should be noted that during the time period t1-t2, the pixel integration value obtained during the time period t0-t1 is read by the pixel reading unit to obtain p1, and p1 can be output. For the output image p2, since the amount of data processing is relatively small and the processing speed is fast in the second processing mode, p2 can be obtained during the time period t1-t2 or during the time period t2-t3 (that is, the encoding unit can work during the time periods t1-t2 and / or t2-t3). This embodiment of the present disclosure does not limit this.

[0112] Figure 5 This is a schematic diagram of the working process of an image processing device provided in an embodiment of the present disclosure, mainly describing the working process of a first processing mode and the working process of obtaining an output image based on the first processing result of the first processing mode.

[0113] like Figure 5 As shown, during the time period t0-t1, the processing module operates based on the first processing mode. In some possible implementations, at time t0, the storage unit is reset to prepare for exposure processing. During the t0-t1 time period, the storage unit sequentially performs long-term exposure on each pixel to obtain the exposure signal (i.e., pixel integral value) of each pixel, and enters a hold state at time t1. After time t1, the pixel integral values ​​are read sequentially (i.e., time-division reading). After reading all pixel integral values, the output image is obtained and transmitted externally. Meanwhile, the t1-t2 time period is a multiplexing time period, used not only to read frame images but also to acquire dynamic visual event data so as to generate event images based on the dynamic visual event data. During the t2-t3 time period, the processing module resumes operation based on the first processing mode, and the working process is similar to that of the t0-t1 time period. During the t3-t4 time period, it is again in a multiplexing state. On the one hand, it reads the integral values ​​of each pixel obtained during the t2-t3 time period to obtain frame images; on the other hand, the photosensitive module acquires dynamic visual event data. The working process can be found in other related content and will not be described in detail here.

[0114] As mentioned above, whether the second processing mode outputs dynamic visual event data is related to factors such as the preset threshold and the sensitivity of the comparison unit. Regarding the preset threshold, if it is set too low, dynamic visual event data may be output even for pixels with small changes in light intensity, resulting in a large data processing volume. Conversely, if the preset threshold is set too high, dynamic visual event data may not be output for some pixels with large changes in light intensity. Therefore, a scientifically reasonable preset threshold needs to be set. Furthermore, if the sensitivity of the comparison unit is low, it cannot accurately and promptly detect changes in light intensity, leading to inaccurate output results or excessively long response times.

[0115] Considering that even if the electrical signal of a single pixel (a single photosensitive sensor) changes, the amount of change is relatively small, the comparison unit connected to it needs to be particularly sensitive to detect this change in time, which leads to high cost. On the other hand, if a comparison unit with relatively low sensitivity is used, it may lead to a longer response time and an increase in latency.

[0116] In view of this, in some possible implementations, the reaction speed of the comparison unit can be improved by merging local pixels, thereby increasing the processing efficiency of the image processing device.

[0117] In some possible implementations, the photosensitive module includes multiple photosensitive sensors, and by connecting multiple photosensitive sensors to the same comparison unit, local pixels can be merged to improve response speed.

[0118] In some possible implementations, the comparison unit is connected to multiple photosensitive sensors, including: the comparison unit is connected to multiple photosensitive sensors corresponding to the same color channel; and / or, the comparison unit is connected to multiple photosensitive sensors in a preset area; and / or, the comparison unit is connected to at least one preset group of photosensitive sensor pixels, each photosensitive sensor pixel group including multiple photosensitive sensors.

[0119] The number of photosensors connected to a comparison unit can be set based on one or more of experience, statistical data, and task requirements. Generally, when high accuracy is required, the number of photosensors connected to a comparison unit can be appropriately reduced to improve accuracy; when high response speed is required, the number of photosensors connected to a comparison unit can be appropriately increased to reduce response time and latency.

[0120] In some alternative implementations, when the comparison unit is connected to multiple photosensitive sensors, the comparison unit determines the signal sum based on the electrical signals of the multiple connected photosensitive sensors, determines the signal difference based on the signal sum and the reference signal value, and determines whether to output dynamic visual event data based on the signal difference and a preset threshold.

[0121] It should be understood that when the comparison unit is connected to a photosensitive sensor, the comparison unit can determine the signal difference based on the electrical signal of the connected photosensitive sensor and the reference signal value, and determine whether to output dynamic visual event data based on the signal difference and a preset threshold.

[0122] It should be noted that the effect of improving the response speed can be achieved by connecting multiple photosensitive sensors to a comparator unit. The above implementation is only an example to illustrate the connection method between the comparator unit and multiple photosensitive sensors. This disclosure does not limit the connection method between the comparator unit and multiple photosensitive sensors.

[0123] It should be understood that in some possible implementations, a comparison unit can also be set for each photosensitive sensor, with each photosensitive sensor connected to the corresponding comparison unit to determine whether the corresponding pixel of the photosensitive sensor needs to output dynamic visual event data.

[0124] The following is combined Figures 6-11 The connection method of the comparison unit in the embodiments of this disclosure will be described in detail.

[0125] Figure 6 This is a schematic diagram illustrating the connection between a comparison unit and a photosensor, provided in an embodiment of this disclosure. (Refer to...) Figure 6 The photosensitive sensor includes three types of sensors corresponding to the R (red) color channel, G (green) color channel, and B (blue) color channel, which are arranged in a preset order to form a pixel array.

[0126] In some possible implementations, the comparison unit is connected to multiple photosensors corresponding to the same color channel. For example... Figure 6 As shown, the four photosensitive sensors corresponding to the R color channel are connected to the same comparison unit. When performing a comparison operation, the comparison unit calculates the signal sum with the electrical signals output by the four photosensitive sensors, compares the signal sum with a reference signal value, obtains the corresponding signal difference, and then determines the duration of the output dynamic visual event data related to these four pixels based on the signal difference and a preset threshold.

[0127] The photosensors corresponding to the G and B color channels can also be connected to the comparator unit in a similar way, which will not be described in detail here.

[0128] Figure 7 This is a schematic diagram illustrating the connection between a comparison unit and a photosensor, provided in an embodiment of this disclosure. (Refer to...) Figure 7 R-G-G-B form a photosensitive sensor pixel group, which includes four photosensitive sensor pixel groups in total, and each photosensitive sensor pixel group is connected to a comparator unit.

[0129] The first comparison unit is connected to four photosensitive sensors in the photosensitive sensor pixel group located in the upper left corner, the second comparison unit is connected to four photosensitive sensors in the photosensitive sensor pixel group located in the upper right corner, the third comparison unit is connected to four photosensitive sensors in the photosensitive sensor pixel group located in the lower left corner, and the fourth comparison unit is connected to four photosensitive sensors in the photosensitive sensor pixel group located in the lower right corner.

[0130] Figure 8 This is a schematic diagram illustrating the connection between a comparison unit and a photosensor, provided in an embodiment of this disclosure. (Refer to...) Figure 8Sixteen photosensitive sensors arranged in a 4x4 array form a preset area, and the photosensitive sensors located within this preset area are connected to the same comparison unit.

[0131] like Figure 8 As shown, the sixteen photosensors are arranged according to... Figure 8 The sensors are connected in series as shown and connected to the comparison unit. When determining whether to output dynamic visual event data, the comparison unit compares the sum of the signal differences from the sixteen sensors with a preset threshold, and then determines whether to perform the output operation based on the comparison result.

[0132] It should be noted that in some possible implementations, multiple preset regions can be set in a pixel array, and different preset regions can have the same or different sizes.

[0133] Figure 9 This is a schematic diagram illustrating the connection between a comparison unit and a photosensor, provided in an embodiment of this disclosure. (Refer to...) Figure 9 Twenty-four photosensitive sensors arranged in a 4x6 array form two preset areas. Sixteen photosensitive sensors arranged in a 4x4 array on the left form one preset area, and eight photosensitive sensors arranged in a 4x2 array on the right form another preset area. Photosensitive sensors located in the same preset area are connected to the same comparison unit.

[0134] like Figure 9 As shown, the twenty-four photosensitive sensors arranged in the left array are connected to the fifth comparison unit, and the eight photosensitive sensors arranged in the right array are connected to the sixth comparison unit.

[0135] It should be noted that a pixel array may use only one connection method or multiple connection methods at the same time, and the embodiments disclosed herein do not impose any restrictions on this.

[0136] Figure 10 This is a schematic diagram illustrating the connection between a comparison unit and a photosensor, provided in an embodiment of this disclosure. (Refer to...) Figure 10 In a 4x4 pixel array, the upper 2x4 portion corresponds to a preset area, and the lower 2x4 portion includes two photosensitive sensor pixel groups.

[0137] like Figure 10 As shown, the eight photosensitive sensors in the preset area are connected to the seventh comparison unit, the lower left photosensitive sensor pixel group is connected to the eighth comparison unit, and the lower right photosensitive sensor pixel group is connected to the ninth comparison unit. Therefore, it can be seen that in... Figure 10 In this process, a connection between the photosensitive sensor and the comparison unit is established based on a preset area and a group of photosensitive sensor pixels.

[0138] In some possible implementations, the photosensor can be divided into two parts: one part operates in a first processing mode, and the other part operates in a second processing mode. This configuration alleviates the workload on the photosensor, and even if the photosensor corresponding to the first processing mode malfunctions, it will not affect the normal operation of the second processing mode; similarly, even if the photosensor corresponding to the second processing mode malfunctions, it will not affect the normal operation of the first processing mode, thus improving the robustness of the image processing device.

[0139] For example, the photosensitive area composed of multiple photosensitive sensors includes at least one first area and at least one second area; wherein the photosensitive sensor in the first area outputs an electrical signal when in a first processing mode, and the photosensitive sensor in the second area outputs an electrical signal when in a second processing mode.

[0140] Figure 11 This is a schematic diagram illustrating the distribution of a photosensor according to an embodiment of this disclosure. (Refer to...) Figure 11 The photosensitive sensor is divided into two first regions and two second regions. The photosensitive sensor in the first region outputs an electrical signal in the first processing mode, and the photosensitive sensor in the second region outputs an electrical signal in the second processing mode.

[0141] Similar to the above, each second region can be connected to a corresponding comparison unit, and multiple second regions can also share a comparison unit, thereby further realizing the merging of local pixels, effectively improving the response speed of the comparison unit and increasing the processing efficiency of the image processing device.

[0142] Figure 12 This is a schematic diagram illustrating the connection between a comparison unit and a photosensor, provided in an embodiment of this disclosure, for the purpose of description. Figure 11 The connection relationship between the photosensor and the comparator unit in the second region shown is illustrated. Figure 12 As shown, the photosensitive sensor in the second area in the upper right corner is connected to the tenth comparison unit, and the photosensitive sensor in the second area in the lower left corner is connected to the eleventh comparison unit.

[0143] Figure 13 This is a schematic diagram illustrating the connection between a comparison unit and a photosensor, provided in an embodiment of this disclosure, for the purpose of description. Figure 11 The connection relationship between the photosensor and the comparator unit in the second region shown is illustrated. Figure 13 As shown, the two photosensitive sensors in the second region are connected to the twelfth comparison unit.

[0144] It should be noted that since the photosensitive sensor in the first area does not operate in the second processing mode and does not need to perform operations such as comparing electrical signals, the photosensitive sensor in the first area does not need to be connected to the comparison unit.

[0145] It should also be noted that in some possible implementations, the image processing device can incorporate two types of sensors, using different types of sensors for different processing modes. For example, the image processing device may include both a frame-based photosensitizer and a dynamic vision sensor. During operation, either sensor can be selected to perform the image processing task according to the processing requirements. While this eliminates the need for a comparison unit, the higher cost of the dynamic vision sensor consequently increases the overall cost of the image processing device.

[0146] Secondly, embodiments of this disclosure provide an image processing method.

[0147] Figure 14 A flowchart illustrating an image processing method provided in an embodiment of this disclosure. (Refer to...) Figure 14 The image processing method includes:

[0148] In step S141, a corresponding electrical signal is generated based on the incident light signal.

[0149] In step S142, the processing mode of the electrical signal in the time dimension is determined, wherein the processing mode includes a first processing mode based on frame vision and a second processing mode based on dynamic vision.

[0150] In step S143, the electrical signal is processed using time-division multiplexing according to the processing mode to obtain the processing result.

[0151] In step S144, the output image is obtained based on the processing result.

[0152] In some possible implementations, in step S141, a corresponding electrical signal can be generated based on the intensity of the incident light signal; wherein the electrical signal includes at least one of a capacitance signal, a voltage signal, and a current signal.

[0153] For example, the light intensity is determined based on the incident light signal, and a corresponding capacitance signal, a corresponding voltage signal, or a corresponding current signal is generated based on the light intensity.

[0154] In some possible implementations, step S142 is used to determine the processing mode of the electrical signal at various time periods or moments, such that, in the time dimension, the processing mode of the electrical signal is manifested as an alternation between a first processing mode and a second processing mode.

[0155] For example, step S142 can control the electrical signal processing mode by gating the processing mode through a time control circuit or by sending a control signal to indicate the processing mode.

[0156] In some possible implementations, in step S143, the electrical signals generated in the first and second cycles can be processed alternately using a first processing mode and a second processing mode based on time-division multiplexing.

[0157] For example, the corresponding processing module can be controlled to process the electrical signal using a first processing mode in the first cycle and a second processing mode in the second cycle.

[0158] In some possible implementations, in step S144, for the first processing result obtained by the first processing mode, the corresponding output image can be obtained through pixel reading operation; for the second processing result obtained by the second processing mode, the corresponding pulse code can be obtained through encoding processing, and the output image can be obtained according to the pulse code.

[0159] It should be noted that after obtaining the output image, various machine vision tasks can be performed based on the output image. The embodiments of this disclosure do not limit the application scenarios of the output image.

[0160] The processing procedures for each step can be found in the description of the relevant content in the embodiments of this disclosure, and will not be described in detail here.

[0161] It should be noted that the above image processing method can be implemented by any of the image processing apparatuses in the embodiments of this disclosure.

[0162] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0163] In addition, this disclosure also provides electronic devices and computer-readable storage media, all of which can be used to implement any of the image processing methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section, and will not be repeated here.

[0164] Figure 15 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0165] Reference Figure 15This disclosure provides an electronic device, which includes: at least one processor 1501; at least one memory 1502; and one or more I / O interfaces 1503 connected between the processor 1501 and the memory 1502; wherein the memory 1502 stores one or more computer programs that can be executed by the at least one processor 1501, and the one or more computer programs are executed by the at least one processor 1501 to enable the at least one processor 1501 to perform the above-described image processing method.

[0166] In some possible implementations, the aforementioned electronic device further includes a photosensitive pixel circuit (corresponding to a photosensitive module) and a control circuit. The photosensitive pixel circuit generates a corresponding electrical signal based on the incident light signal, and the control circuit controls the processing mode of the electrical signal. The processor, corresponding to a processing module, processes the electrical signal according to the processing mode determined by the control circuit and obtains an output image based on the processing result. During the above processing, data storage and caching operations may be required. A memory can be used to implement this function, for example, storing pixel integration values, dynamic visual event data, and output image data. Furthermore, the connections between the various functional units can be achieved through I / O interfaces to ensure data transmission between the functional units, thereby guaranteeing the smooth execution of the image processing task.

[0167] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the image processing method described above. The computer-readable storage medium may be volatile or non-volatile.

[0168] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described image processing method.

[0169] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0170] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0171] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0172] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0173] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0174] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0175] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0176] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0177] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0178] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. An image processing apparatus characterized by comprising: include: The photosensitive module is used to generate a corresponding electrical signal based on the incident light signal; The control module is used to control the processing mode of the electrical signal generated by the photosensitive module in a time-division multiplexing manner; The processing module is used to process the electrical signal according to the processing mode determined by the control module to obtain the processing result; The processing modes include a first processing mode based on frame vision and a second processing mode based on dynamic vision. The processing module adopts the first processing mode in each first cycle and the second processing mode in each second cycle, and the first cycle and the second cycle are alternately set in the time dimension.

2. The image processing apparatus according to claim 1, characterized by, The processing module includes a storage unit and a comparison unit; The control module is used to control the connection between the photosensitive module and the storage unit when it is determined that the first processing mode is to be used, so as to obtain the first processing result; And / or, The control module is used to control the connection between the photosensitive module and the comparison unit to obtain the second processing result when it is determined that the second processing mode is to be used.

3. The image processing apparatus according to claim 2, characterized by, For the electrical signals generated by the photosensitive module in the first and second cycles, the control module controls the processing module to alternately process them using the first processing mode and the second processing mode.

4. The image processing apparatus according to claim 3, characterized by During the first cycle, the control module controls the storage unit to connect with the photosensitive module to obtain frame data; During the second cycle, the control module controls the comparison unit to connect with the photosensitive module, and determines whether to output dynamic visual event data based on the comparison result of the comparison unit.

5. The image processing apparatus according to claim 4, characterized by The comparison unit operates based on a preset time step. At the i-th time step, the comparison unit determines the signal difference based on the electrical signal output by the photosensitive module at the i-th time step and the electrical signal at the (i-1)-th time step, and if the signal difference is greater than a preset threshold, determines that the second processing result is to output dynamic visual event data.

6. The image processing apparatus according to claim 5, characterized by The photosensitive module includes multiple photosensitive sensors, which are used to generate corresponding electrical signals based on the incident light signal; the comparison unit is connected to at least one photosensitive sensor.

7. The image processing apparatus according to claim 6, characterized by The comparison unit is connected to multiple photosensitive sensors, including: The comparison unit is connected to multiple photosensitive sensors corresponding to the same color channel; And / or, The comparison unit is connected to multiple photosensitive sensors in a preset area; And / or, The comparison unit is connected to at least one preset group of photosensitive sensor pixels, and each photosensitive sensor pixel group includes multiple photosensitive sensors.

8. The image processing apparatus according to claim 4, characterized by, The processing module further includes a pixel reading unit, and the pixel reading unit is connected to the storage unit; The pixel reading unit is used to read data stored in the storage unit in the previous first cycle adjacent to the second cycle in the second cycle.

9. The image processing apparatus according to claim 4, characterized by, The processing module further includes an encoding unit, and the encoding unit is connected to the comparison unit; The encoding unit is used to encode the dynamic visual event data output by the comparison unit to obtain a pulse code, and to obtain an output image based on the pulse code.

10. The image processing apparatus according to claim 1, wherein The photosensitive module includes multiple photosensitive sensors; The photosensitive sensor is configured to generate an electrical signal corresponding to an intensity of an incident light signal. The electrical signal includes at least one of a capacitance signal, a voltage signal, and a current signal.

11. The image processing apparatus according to claim 10, characterized by The photosensitive sensor includes at least one of a first sensor, a second sensor, and a third sensor. The first sensor is configured to obtain a light intensity of a light signal and generate a corresponding capacitance signal based on the light intensity, the second sensor is configured to obtain a light intensity of a light signal and generate a corresponding voltage signal based on the light intensity, and the third sensor is configured to obtain a light intensity of a light signal and generate a corresponding current signal based on the light intensity.

12. The image processing apparatus according to claim 10, wherein The photosensitive region composed of a plurality of photosensitive sensors includes at least one first region and at least one second region. The photosensitive sensors in the first region output electrical signals in a first processing mode, and the photosensitive sensors in the second region output electrical signals in a second processing mode.

13. An image processing method, characterized by, The method for the image processing device according to any one of claims 1-12, the method comprising: generating an electrical signal corresponding to an incident light signal; determining a processing mode of the electrical signal in a time dimension; processing the electrical signal in a time division multiplexing manner according to the processing mode to obtain a processing result; obtaining an output image according to the processing result; wherein the processing mode includes a first processing mode based on frame vision and a second processing mode based on dynamic vision.

Images