Image sensor-based data processing method and apparatus, image processing system

Abstract

The present disclosure provides a data processing method and apparatus based on an image sensor, and an image processing system, which belong to the computer technology field. The data processing method includes: acquiring spatio-temporal signals of a plurality of pixels through an image sensor; grouping the plurality of pixels to obtain a plurality of pixel groups; determining group addresses for each pixel group and macro addresses for the plurality of pixel groups based on pixel addresses within each pixel group; and encoding the signal packets for the plurality of pixel groups based on the sampling period corresponding to the spatio-temporal signals, the group addresses for each pixel group, the macro addresses for the plurality of pixel groups, and the spatio-temporal signals of the pixels within each pixel group. According to the embodiments of the present disclosure, it is possible to reduce encoding redundancy while well preserving pixel signal information.

Description

[Technical Field]

[0001] TECHNICAL FIELD Embodiments of the present disclosure relate to the field of computer technology, and in particular to an image sensor-based data processing method and apparatus, an image processing system, an electronic device, and a computer-readable storage medium. [Background technology]

[0002] An event camera, also known as a dynamic vision sensor (DVS), is a new type of imaging system. Unlike conventional cameras, which use a shutter to control the frame rate and record the light intensity for every pixel in each frame, an event camera is sensitive to the rate of change of light intensity, with each pixel individually recording the change in the logarithmic value of the light intensity at that pixel, and generating a forward or negative pulse after the change exceeds a threshold. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure provides an image sensor-based data processing method and apparatus, an image processing system, an electronic device, and a computer-readable storage medium. [Means for solving the problem]

[0004] In a first aspect, the present disclosure provides a data processing method based on an image sensor, including: acquiring spatiotemporal signals of a plurality of pixels via an image sensor; grouping the plurality of pixels to obtain a plurality of pixel groups; determining a group address for each of the pixel groups and a macro address for the plurality of pixel groups based on pixel addresses within each of the pixel groups; and encoding based on a sampling period corresponding to the spatiotemporal signals, the group address for each of the pixel groups, the macro addresses for the plurality of pixel groups, and the spatiotemporal signals of pixels within each of the pixel groups to obtain signal packets for the plurality of pixel groups.

[0005] In a second aspect, the present disclosure provides a data processing device based on an image sensor, including: an acquisition module for acquiring spatiotemporal signals of a plurality of pixels via an image sensor; a grouping module for grouping the plurality of pixels to obtain a plurality of pixel groups; a determination module for determining a group address of each of the pixel groups and a macro address of the plurality of pixel groups based on a pixel address within each of the pixel groups; and an encoding module for encoding based on a sampling period corresponding to the spatiotemporal signals, the group address of each of the pixel groups, the macro addresses of the plurality of pixel groups, and the spatiotemporal signals of pixels in each of the pixel groups to obtain signal packets of the plurality of pixel groups.

[0006] In a third aspect, the present disclosure provides an image processing system including an image sensor-based data processing device and at least one image sensor, wherein the image sensor is used to acquire spatiotemporal signals of a plurality of pixels based on a preset sampling period, and the image sensor-based data processing device is used to perform the image sensor-based data processing method described in any one of the embodiments of the present disclosure.

[0007] In a fourth aspect, the present disclosure provides an electronic device comprising at least one processor and a memory communicatively coupled to the at least one processor, the memory storing one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor such that the at least one processor can perform the image sensor-based data processing method described above.

[0008] In a fifth aspect, the present disclosure provides an electronic device including a plurality of processing cores and an on-chip network configured to interact with data between said plurality of processing cores and with external data, wherein one or more instructions are stored in one or more of said processing cores, and wherein said one or more instructions are executed by said one or more processing cores to enable said one or more processing cores to perform the image sensor-based data processing method described above.

[0009] In a sixth aspect, the present disclosure provides a computer-readable storage medium having stored thereon a computer program that, when executed by a processor / processing core, is capable of performing the image sensor-based data processing method described above.

[0010] In a seventh aspect, an embodiment of the present disclosure provides a computer program product including computer readable code, or a non-volatile computer readable storage medium having computer readable code thereon, which, when executed by a processor of an electronic device, causes the processor in the electronic device to perform the image sensor-based data processing method described above. [Effects of the Invention]

[0011] The embodiments provided by the present disclosure acquire spatiotemporal signals of multiple pixels via an image sensor, and through the spatiotemporal signals, not only can determine the signal change status of the pixel itself in the time dimension, but also can determine the signal difference status between the current pixel and adjacent pixels, thereby more comprehensively reflecting the signal information of the pixel. Next, multiple pixels are grouped to obtain multiple pixel groups. Taking into full consideration the locality of the spatiotemporal signal changes of the pixels, multiple pixels with similar changes can be grouped into one pixel group through grouping, thereby reducing the redundancy of encoding address information for each change event during subsequent encoding. Based on the pixel addresses within each pixel group, group addresses for each pixel group and macro addresses for multiple pixel groups are determined. During subsequent encoding, the macro addresses and group addresses are encoded with a larger granularity than individual pixel addresses, thereby reducing the redundancy of address encoding. Finally, encoding is performed based on the sampling period corresponding to the spatio-temporal signal, the group address of each pixel group, the macro address of the plurality of pixel groups, and the spatio-temporal signal of the pixels in each pixel group, thereby obtaining signal packets of the plurality of pixel groups. This reduces the time redundancy and address redundancy of the encoding, while retaining the signal information of each pixel. Thus, the corresponding image can be restored based on the signal packets, and various tasks can be performed based on the image.

[0012] It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory and are not restrictive of the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the drawings. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a conceptual diagram of an encoding method for an event camera provided by the related art. [Figure 2] 1 is a flowchart of an image sensor-based data processing method provided by an embodiment of the present disclosure. [Figure 3] FIG. 2 is a conceptual diagram of a pixel distribution corresponding to an image sensor provided by an embodiment of the present disclosure. [Figure 4] FIG. 1 is a conceptual diagram of a sampling operation procedure based on an image sensor provided by an embodiment of the present disclosure. [Figure 5] FIG. 1 is a conceptual diagram of a sampling operation procedure based on an image sensor provided by an embodiment of the present disclosure. [Figure 6] FIG. 1 is a conceptual diagram of pixel grouping provided by an embodiment of the present disclosure. [Figure 7] FIG. 1 is a conceptual diagram of pixel grouping provided by an embodiment of the present disclosure. [Figure 8] FIG. 2 is a conceptual diagram of a signal packet provided by an embodiment of the present disclosure. [Figure 9] FIG. 1 is a block diagram of an image sensor-based data processing device provided by an embodiment of the present disclosure. [Figure 10] FIG. 1 is a block diagram of an image processing system provided by an embodiment of the present disclosure. [Figure 11] FIG. 1 is a block diagram of an electronic device provided by an embodiment of the present disclosure. [Figure 12] FIG. 1 is a block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] The present disclosure will be described in more detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present disclosure and are not intended to limit the present disclosure. In addition, for ease of explanation, the drawings show only parts relevant to the present disclosure, rather than the entire configuration.

[0015] Both DVS and DAVIS cameras use asynchronous sampling and output methods, so signals are usually encoded and output externally based on Address Event Representation (AER), making the signal sampling and output method difficult to be compatible with common output protocols.

[0016] 1 is a conceptual diagram of an event camera encoding system provided by the related art. Referring to FIG. 1, a pixel located at address (x,y) (e.g., a pixel located at address (1,1)) first issues a pulse signal and polarity request to an arbiter. If the arbiter approves the signal request, an address encoder and handshaking logic encode the pixel's address information, pulse signal polarity, and current timestamp, and issue the encoded data as an AER packet to an out-of-slice receiver. Here, the pulse signal polarity indicates whether the pixel brightness increases or decreases compared to the previous sampling.

[0017] Considering the above, first, whether it is a DVS camera or a DAVIS camera, both use an asynchronous AER output protocol. This asynchronous method is difficult to be compatible with commonly used data receiving and processing chips, and processing must be done by a dedicated asynchronous chip (which may be an on-chip chip or an off-chip chip). Second, in terms of encoding method and efficiency, the AER packet format encodes each pixel, and only one data bit in each packet characterizes the actual pixel value, while the other bits characterize the abscissa, ordinate, and timestamp, resulting in a large amount of data bit waste. Furthermore, the AER packet format encodes a time pulse signal, making it difficult to reflect pixel signal information from the spatial dimension.

[0018] In light of this, the embodiments of the present disclosure provide a data processing method and apparatus based on an image sensor, and an image processing system, which can solve at least one of the above technical problems.

[0019] The embodiments provided by the present disclosure acquire spatiotemporal signals of multiple pixels via an image sensor, and through the spatiotemporal signals, not only can determine the signal change status of the pixel itself in the time dimension, but also can determine the signal difference status between the current pixel and adjacent pixels, thereby more comprehensively reflecting the signal information of the pixel. Next, multiple pixels are grouped to obtain multiple pixel groups. Taking into full consideration the locality of the spatiotemporal signal changes of the pixels, multiple pixels with similar changes can be grouped into one pixel group through grouping, thereby reducing the redundancy of encoding address information for each change event during subsequent encoding. Based on the pixel addresses within each pixel group, group addresses for each pixel group and macro addresses for multiple pixel groups are determined. During subsequent encoding, the macro addresses and group addresses are encoded with a larger granularity than individual pixel addresses, thereby reducing the redundancy of address encoding. Finally, encoding is performed based on the sampling period corresponding to the spatio-temporal signal, the group address of each pixel group, the macro address of the plurality of pixel groups, and the spatio-temporal signal of the pixels in each pixel group, thereby obtaining signal packets of the plurality of pixel groups. This reduces the time redundancy and address redundancy of the encoding, while retaining the signal information of each pixel. Thus, the corresponding image can be restored based on the signal packets, and various tasks can be performed based on the image.

[0020] The image sensor-based data processing method according to the embodiment of the present disclosure may be performed by an electronic device such as a terminal device or a server. The terminal device may be a user device (User Equipment, UE), a mobile device, a user terminal, a terminal, a mobile phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc., and the method may be implemented by a processor calling computer-readable program instructions stored in a memory. The server may be an independent physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing.

[0021] In a first aspect, embodiments of the present disclosure provide a data processing method based on an image sensor.

[0022] 2 is a flowchart of a data processing method based on an image sensor provided by an embodiment of the present disclosure. Referring to FIG. 2, the data processing method includes the following steps:

[0023] In step S21, spatiotemporal signals of a plurality of pixels are acquired via an image sensor.

[0024] In step S22, a plurality of pixels are grouped to obtain a plurality of pixel groups.

[0025] In step S23, a group address for each pixel group and macro addresses for a plurality of pixel groups are determined based on the pixel addresses within each pixel group.

[0026] In step S24, signal packets of the plurality of pixel groups are obtained by encoding based on the sampling period corresponding to the spatio-temporal signal, the group address of each pixel group, the macro address of the plurality of pixel groups, and the spatio-temporal signal of the pixels in each pixel group.

[0027] According to an embodiment of the present disclosure, spatiotemporal signals of multiple pixels are acquired via an image sensor. The spatiotemporal signals can be used to determine not only the signal change status of the pixel itself in the time dimension, but also the signal difference status between the current pixel and its neighboring pixels, thereby more comprehensively reflecting the signal information of the pixel. Next, the multiple pixels are grouped to obtain multiple pixel groups. Taking into full consideration the locality of the spatiotemporal signal changes of the pixels, multiple pixels with similar changes can be grouped into one pixel group. This reduces the redundancy of encoding address information for each change event during subsequent encoding. Based on the pixel addresses within each pixel group, group addresses for each pixel group and macro addresses for multiple pixel groups are determined. During subsequent encoding, the macro addresses and group addresses are coded with a larger granularity than individual pixel addresses, thereby reducing the redundancy of address coding. Finally, encoding is performed based on the sampling period corresponding to the spatio-temporal signal, the group address of each pixel group, the macro address of the plurality of pixel groups, and the spatio-temporal signal of the pixels in each pixel group, thereby obtaining signal packets of the plurality of pixel groups. This reduces the time redundancy and address redundancy of the encoding, while retaining the signal information of each pixel. Thus, the corresponding image can be restored based on the signal packets, and various tasks can be performed based on the image.

[0028] In some preferred embodiments, in step S21, the image sensor includes a signal sensor capable of sensing light intensity, and one image sensor can correspond to multiple pixels, which can be arranged in various ways to form a corresponding pixel array, and the spatiotemporal signals of these pixel arrays can obtain a corresponding image.

[0029] In some preferred embodiments, the spatiotemporal signal is a light intensity-based signal, and the spatiotemporal signal may be used at least to characterize the signal change information of a pixel in the time dimension and the signal difference information in the space dimension, i.e., the spatiotemporal signal of a pixel can reflect the signal information of the pixel from both the time dimension and the space dimension.

[0030] In some preferred embodiments, the image sensor can also be used to obtain color signals of multiple pixels, where the color signals of the pixels can reflect the color information of the pixels, and the color signals of the multiple pixels can obtain color pixels with color distribution.

[0031] In some preferred embodiments, after irradiating a light beam onto the surface of an object to be imaged and transmitting it through an optical path such as reflection, refraction, etc., the image sensor can capture an original signal (which may be related to signal processing such as, but not limited to, photoelectric conversion), and decompose this original signal into a luminance signal Y and a chrominance signal, where the luminance signal can be used to determine a spatiotemporal signal, and the chrominance signal can be re-decomposed to obtain color difference signals U and V, and by performing matrix operation on the luminance signal Y, the color difference signals U and V, a color signal - RGB signal - can be obtained.

[0032] In some preferred embodiments, two types of image sensors may be provided, one used to acquire the light intensity signal of the pixel to obtain the spatiotemporal signal of the pixel, and the other used to acquire the color signal of the pixel.

[0033] 3 is a conceptual diagram of a pixel distribution corresponding to an image sensor provided by an embodiment of the present disclosure. The image sensor includes two types: one type is used to acquire the light intensity signal of the pixel, and the other type is used to acquire the color signal of the pixel. Correspondingly, the pixels as the light-sensitive units of the image sensor are also divided into two types: one type is a light intensity pixel used to capture the light intensity signal, and the other type is a color pixel used to capture the color signal. Here, after obtaining the light intensity signal through the light intensity pixel, the corresponding spatiotemporal signal can be obtained through further processing.

[0034] As shown in Figure 3, the light intensity pixels and color pixels are arranged in a cross-sectional layout in both the row and column directions, with four color pixels distributed around one light intensity pixel, and four light intensity pixels (excluding edge pixels) distributed around one color pixel. After obtaining a light intensity signal through the light intensity pixels, the corresponding spatiotemporal signal is obtained through further processing. In this configuration, the light intensity pixels and color pixels are distributed uniformly, so the resolution of the image obtained based on the spatiotemporal signal is close to that of the image obtained based on the color signal.

[0035] The above distribution of light intensity pixels and color pixels is merely an example, and light intensity pixels and color pixels can be set using other distribution methods. For example, light intensity pixels and color pixels can be set using a row and column spacing method. For example, in a target area, a number of pixels can be randomly selected as light intensity pixel regions, and the remaining region can be set as color regions. For example, the target area can be divided into luminance-sensitive regions and color-sensitive regions in advance, and more light intensity pixels can be set in the luminance-sensitive regions, and more color pixels can be set in the color-sensitive regions. Furthermore, the distribution of light intensity pixels and color pixels can be determined based on experience, statistical data, simulation data, etc., and embodiments of the present disclosure are not limited thereto.

[0036] In other words, in some preferred embodiments, the image sensor in the embodiments of the present disclosure is a dual-mode sensor, and based on the dual-mode sensor, light intensity pixels and color pixels of the imaged object can be obtained, where the light intensity pixels can represent the changing conditions of the object from the time dimension and the space dimension, and the color pixels can display the object from the color angle, thereby realizing the effect of representing object information from more dimensions through the dual-mode sensor, and facilitating image analysis from multiple angles.

[0037] In some preferred embodiments, the spatiotemporal signals of the pixels are generated and output based on an event trigger scheme. In other words, the pixels are operated using an asynchronous sampling and output scheme, and if the signal change intensity of a pixel exceeds a preset threshold, a spatiotemporal signal is output to the outside. Here, the signal change intensity includes the signal change intensity in the time dimension and / or the signal change intensity in the space dimension.

[0038] For example, if the difference value between the light intensity of a pixel at the current time and the light intensity at the immediately previous time is greater than a preset threshold, one spatiotemporal signal can be output to the outside.

[0039] For example, if the difference between the light intensity of a pixel at a current time and the light intensity of the surrounding adjacent pixels is a first difference, and the difference between the light intensity of the pixel at the previous time and the light intensity of the periodic pixel is a second difference, and the difference between the first difference and the second difference is greater than a preset threshold, one spatiotemporal signal can be output to the outside.

[0040] In some preferred embodiments, the spatiotemporal signals of the plurality of pixels are acquired based on a unified sampling period, in other words, the spatiotemporal signals of the plurality of pixels are collected in a synchronous manner, where the sampling period is determined based on any one or more of experience, statistical data, simulation results, and imaging needs, and the embodiments of the present disclosure are not limited thereto.

[0041] In some preferred embodiments, the step of acquiring spatiotemporal signals of the plurality of pixels through the image sensor based on the preset sampling period includes the step of sampling output data of the image sensor based on the sampling period to obtain spatiotemporal signals of the plurality of pixels corresponding to the image sensor. In this embodiment, there is no need to pay too much attention to the operating frequency of the image sensor itself, and it is only necessary to sample its output data based on the sampling period, so there is no need to change or update the operating frequency of the image sensor.

[0042] In some preferred embodiments, the step of acquiring the spatiotemporal signals of the plurality of pixels via the image sensor based on the preset sampling period includes, when the image sensor outputs data to the outside based on the sampling period, obtaining the corresponding spatiotemporal signals of the plurality of pixels based on output data of the image sensor. In this embodiment, since the image sensor operates at the sampling period, the spatiotemporal signals of the plurality of pixels can be obtained directly based on the output data of the image sensor.

[0043] Based on the above, the image sensor achieves globally synchronized sampling output by recording the change in the time dimension and the difference in the spatial dimension for each pixel using a unified time step (corresponding to the sampling period).

[0044] In addition, in the related art, each pixel point of the event camera operates independently and asynchronously, and therefore signals of multiple pixels cannot be collected and output based on a unified sampling period. When operating using an asynchronous sampling method, the sampling time of each pixel is relatively independent, and corresponding spatiotemporal signals must be received and processed based on an asynchronous chip. When operating based on a unified sampling period, spatiotemporal signals of multiple pixels can be acquired in a unified manner, which can support conventional data receiving chips and data processing chips. Since multiple pixels have the same sampling period, a global time encoding method can be adopted for multiple pixels when subsequently encoding the time dimension, eliminating the need for independent encoding in the time dimension for each pixel, and reducing the complexity and redundancy of the time dimension encoding.

[0045] Illustratively, the spatiotemporal signal includes a light intensity-based signal, and the spatiotemporal signal includes at least a time dimension change amount and a space dimension difference amount, where the time dimension change amount is a change amount between the light intensity of the pixel in a current sampling period relative to the light intensity of the pixel in a previous sampling period, and the space dimension difference amount is a difference amount between the light intensity of the pixel in the current sampling period and the light intensity of at least one adjacent pixel, where the adjacent pixel includes one or more pixels positionally adjacent to the current pixel.

[0046] For example, the spatio-temporal signal can further include a light intensity amount in addition to the time dimension change amount and the space dimension difference amount, where the light intensity amount is used to reflect the light intensity of the pixel in the current sampling period. The current spatio-temporal signal can not only obtain the change situation of the pixel and the difference situation with the adjacent pixel, but also the absolute light intensity situation of the pixel, which is equivalent to combining a conventional camera and an event camera based on a dual-mode sensor.

[0047] 4 is a conceptual diagram of a sampling operation procedure based on an image sensor provided by an embodiment of the present disclosure. Referring to FIG. 4, a plurality of pixels (e.g., R1_0, R1_1, ..., R3_n1) are arranged in an array to form a 3*(n1+1) pixel array (n1 is an integer equal to or greater than 1). The spatiotemporal signals of each pixel in the pixel array are acquired under the trigger of a trigger pulse. Here, the trigger pulse outputs a preset sampling period as a time step length, and the spatiotemporal signals of the plurality of pixels are acquired in response to the trigger pulse.

[0048] Taking pixel R2_2 as an example, for the i-th sampling period (i≧1), the spatiotemporal signal includes a time dimension change amount and a space dimension difference amount. Here, the time dimension change amount is the light intensity G 2_2 (i) and the light intensity G in the i-1th sampling period 2_2 The neighboring pixels of pixel R2_2 are the eight pixels surrounding it, which are R1_1, R1_2, R1_3, R2_1, R2_3, R3_1, R3_2, and R3_3, respectively. Correspondingly, there are eight spatial dimension difference amounts, which are the light intensity G 2_2 (i) and the light intensity G of pixel R1_1 in the i-th sampling period 1_1 (i) is the difference between the light intensity G 2_2 (i) and the light intensity G of pixel R1_2 in the i-th sampling period 1_2 (i) is the difference between the light intensity G 2_2 (i) and the light intensity G of pixel R1_3 in the i-th sampling period 1_3 (i) is the difference between the light intensity G 2_2 (i) and the light intensity G of pixel R2_1 at the i-th sampling period 2_1 (i) is the difference between the light intensity G 2_2(i) and the light intensity G of pixel R2_3 at the i-th sampling period 2_3 (i) is the difference between the light intensity G 2_2 (i) and the light intensity G of pixel R3_1 in the i-th sampling period 3_1 (i) is the difference between the light intensity G 2_2 (i) and the light intensity G of pixel R3_2 in the i-th sampling period 3_2 (i) is the difference between the light intensity G 2_2 (i) and the light intensity G of pixel R3_3 in the i-th sampling period 3_3 (i) is the difference between

[0049] 5 is a conceptual diagram of a sampling operation procedure based on the image sensor provided by the present disclosure. Referring to FIG. 5, a plurality of pixels are arranged alternately in an array to form a 3*(n2+1) pixel array, and the spatiotemporal signal of each pixel in the pixel array is collected under the trigger of a trigger pulse.

[0050] Taking pixel R2_0 as an example, for the i-th sampling period, the spatiotemporal signal includes a time dimension change amount and a space dimension difference amount. Here, the time dimension change amount is the light intensity G 2_0 (i) and the light intensity G in the i-1th sampling period 2_0 The adjacent pixels of pixel R2_0 are the four pixels surrounding it, namely R1_0, R1_1, R3_0, and R3_. Correspondingly, the spatial dimension difference amount includes four, which are the light intensity G 2_0 (i) and the light intensity G of pixel R1_0 at the i-th sampling period 1_0 (i) is the difference between the light intensity G 2_0 (i) and the light intensity G of pixel R1_1 in the i-th sampling period 1_1(i) is the difference between the light intensity G 2_0 (i) and the light intensity G of pixel R3_0 at the i-th sampling period 3_0 (i) is the difference between the light intensity G 2_0 (i) and the light intensity G of pixel R3_1 in the i-th sampling period 3_1 (i) is the difference between

[0051] Note that the above is merely an example of neighboring pixels, and other numbers and positions of pixels can be determined as neighboring pixels in other embodiments. For example, for pixel R2_2 in FIG. 4, one or more of the eight neighboring pixels can be arbitrarily selected, and the corresponding spatial dimension difference amount can be determined based on the selected neighboring pixels (e.g., R1_1, R1_3, R3_1, and R3_3 can be selected as neighboring pixels for determining the spatial dimension difference amount). Alternatively, a wider range of pixels can be selected as neighboring pixels (e.g., in addition to the eight neighboring pixels, R1_0, R2_0, and R3_0 can be selected as neighboring pixels of R2_2). The embodiments of the present disclosure are not limited thereto.

[0052] In addition, in the embodiment of the present disclosure, independent data precision is adopted between the time dimension change quantity and the space dimension difference quantity in the space-time signal, so that the data processing method not only supports globally synchronized sampling output, but also supports multiple data precision types for the sampling output.

[0053] In some preferred embodiments, the time dimension change amount has a first preset data precision, and the space dimension difference amount has a second preset data precision, where the first preset data precision corresponds to the same data precision as the second preset data precision, or the first preset data precision corresponds to a data precision different from the second preset data precision. That is, the data precisions of the time dimension change amount and the space dimension difference amount are independent, and the data precisions of both may be the same or different, and the embodiments of the present disclosure are not limited thereto.

[0054] In addition, since the time dimension change amount and the space dimension difference amount have independent data precision, various methods can be adopted when encoding them to construct a packet. For example, when the data precision of the two is different, each of them can be encoded separately to obtain a separate packet. For example, when the data precision of the two is the same, both can be encoded in the same packet. Based on the independent data precision setting method, the flexibility of encoding can be increased.

[0055] In some preferred embodiments, after obtaining the spatiotemporal signals of the plurality of pixels, in step S22, the plurality of pixels are grouped to obtain a plurality of pixel groups, where each pixel group includes at least one pixel, and the number of pixels in the plurality of pixel groups may be the same or different, and the embodiments of the present disclosure are not limited thereto.

[0056] In some preferred embodiments, grouping pixels to obtain pixel groups includes dividing a pixel bar consisting of adjacent pixel groups in an array into one pixel group to obtain the pixel groups, and / or dividing a pixel block consisting of adjacent pixel groups in the arrays into one pixel group to obtain the pixel groups, where if the array includes rows or columns, an array of pixels may refer to a row of pixels or a column of pixels.

[0057] In other words, in the embodiment of the present disclosure, when pixels that are close to each other are considered, their signal change situations are relatively similar, so the pixels are grouped and multiple pixels that are close to each other are divided into one pixel group. When grouping, a pixel bar consisting of multiple adjacent pixel groups may be regarded as one pixel group, using a row or column as a unit, or a pixel block consisting of multiple pixel groups may be regarded as one pixel group, using multiple rows or columns as a unit of pixels.

[0058] Exemplarily, a plurality of pixel groups are composed of an n*m pixel array (n≧1, m≧1, and n and m are not 1 simultaneously). When grouping the pixel array, for the i-th row of pixels (i≦n), the first pixel to the k-th pixel can be divided into one pixel group (1<k<m), and the (k + 1)-th pixel to the 2k-th pixel can be divided into one pixel group. Until all the pixels in the row are divided into corresponding pixel groups (the number of pixels in the last pixel group may be less than or equal to k), the same method is used for the following rows. In this way, the grouping of the pixel array is realized, and a plurality of pixel groups are obtained.

[0059] Exemplarily, a plurality of pixel groups are composed of an n*m pixel array. For the pixels in the first row to the h-th row (h<n), by setting h*k as the size of one pixel group, it is divided into a plurality of pixel groups. For the pixels in the (h + 1)-th row to the 2h-th row, the above operation is repeated until all the pixels are divided into corresponding pixel groups.

[0060] In some preferred embodiments, the step of grouping a plurality of pixels to obtain a plurality of pixel groups includes: determining the number of pixels in each pixel group based on the data accuracy of the spatio-temporal signal and the format requirements of the signal packet; determining the grouping size corresponding to the vector format and / or matrix format, which is used to characterize the data size of a plurality of pixels in the pixel group based on the pixel data in each pixel group; and grouping a plurality of pixels based on the grouping size to obtain a plurality of pixel groups.

[0061] For example, if the signal packet format requirements are that the data occupancy bits are 24 bits and the data precision of the spatio-temporal data is 2 bits, it is determined that up to 12 pixels can be set in one pixel group, and based on this, the grouping size is determined to be one or more of 1*12, 12*1, 2*6, 6*2, 3*4, 4*3, etc., and multiple pixels are grouped based on the determined grouping size to obtain multiple pixel groups. For example, if the pixel size is determined to be 12*1, the 1st to 12th pixels in the first row of pixels are divided into one pixel group, and the 13th to 24th pixels are divided into one pixel group, and the above operation is repeated until all pixels are divided into corresponding pixel groups.

[0062] 6 is a conceptual diagram of pixel grouping provided by the present disclosure. Referring to FIG. 6, a plurality of pixel groups are composed of m*160 pixel arrays, and the format requirements of the signal packet are that the data occupation bit is 24 bits and the data precision of the spatio-temporal data is 2 bits, and it is determined that up to 12 pixels can be set in one pixel group.

[0063] As shown in Figure 6, a pixel group is a pixel bar consisting of 12 adjacent pixel groups in a pixel row unit. In terms of each pixel row, it can be divided into 14 groups, where the first to thirteenth pixel groups each contain 12 pixels, and the fourteenth pixel group contains only four pixels.

[0064] 7 is a conceptual diagram of pixel grouping provided by the present disclosure. Referring to FIG. 7, a plurality of pixel groups are composed of m*160 pixel arrays, and the format requirements of the signal packet determine that, when the data occupation bit is 24 bits and the data precision of the spatio-temporal data is 2 bits, up to 12 pixels can be set in one pixel group.

[0065] As shown in FIG. 7, the grouping size is 3*4, and it is determined that the pixel array is divided into multiple pixel groups based on the grouping size. For example, for the first to third rows of pixels, the first four pixels of the first row, the first four pixels of the second row, and the first four pixels of the third row are divided into one group, thereby obtaining the first pixel group in the first to third rows. Similarly, the remaining pixels are grouped accordingly, thereby obtaining the second to fortieth pixel groups. By performing a similar process on the pixels of the fourth to m-th rows, the grouping process for the entire pixel array is completed, and multiple pixel groups are obtained.

[0066] In some preferred embodiments, one grouping size may be used for a partial region of the pixel array, and another grouping size may be used for other regions, or grouping may be performed in an irregular shape; embodiments of the present disclosure are not limited to this. For example, for the above 160*m pixel array, the grouping scheme shown in Figure 6 may be used for the first to third rows, and the grouping scheme shown in Figure 7 may be used for the remaining pixel region, so that some of the resulting pixel groups correspond to the pixel bar format and some correspond to the pixel block format.

[0067] In some preferred embodiments, after grouping multiple pixels to obtain multiple pixel groups, in order to facilitate subsequent encoding of pixel addresses, in step S23, a group address for each pixel group and a macro address for the multiple pixel groups are determined based on the addresses of the pixels within each pixel group, where the macro addresses for the multiple pixel groups are determined based on the array addresses of the pixel arrays occupied by the multiple pixel groups, and characterize the overall addresses of the multiple pixel groups.

[0068] Taking the pixel group shown in FIG. 6 as an example, if the address of the pixel array in the i-th row is add(i) and the address of the pixel in the i-th row and j-th column is add(i,j), the group address of the first pixel group is add(1,1)-add(1,12) (i.e., add(1,1)-add(1,12) as a whole is one group address, which characterizes the 1st to 12th pixels (R1_0 to R1_11) in the first row corresponding to that pixel group), and the group address of the second pixel group is add(1,13)-add(1,24). (i.e., add(1,13)-add(1,24) as a whole is one group address that characterizes the 13th to 24th pixels (R1_12 to R1_23) in the first row that correspond to that pixel group), and so on to determine the group addresses for each of the remaining pixel groups, where the 14th pixel group contains only four pixels and its group address is add(1,157)-add(1,160), indicating that it corresponds to the 157th to 160th pixels in the first row.

[0069] Furthermore, since the first pixel group to the fourteenth pixel group occupy the pixels in the first column, the macro address of the first pixel group to the fourteenth pixel group is determined to be add(1) (i.e., indicating the address corresponding to the first row pixel), and similarly, macro addresses corresponding to other multiple pixel groups can be determined.

[0070] In the related art, pixels are not grouped, so address encoding needs to be performed on a single pixel, and group addresses for pixel groups and macro addresses for multiple pixel groups are not determined, and there is no need to determine them. In the embodiment of the present disclosure, since grouping processing is performed on pixels, after the group addresses and macro addresses are determined, when encoding is performed in step S24, encoding can be performed using the group addresses and macro addresses in the address dimension, eliminating the need to encode the address for each pixel individually and reducing the redundancy of address encoding.

[0071] In some preferred embodiments, the signal packet includes a timestamp packet, an address packet, and a space-time packet, and correspondingly, the step of encoding based on the sampling period corresponding to the space-time signal, the group address of each pixel group, the macro address of the plurality of pixel groups, and the space-time signal of the pixels in each pixel group to obtain signal packets of the plurality of pixel groups includes: determining timestamps for the plurality of pixel groups based on the sampling period corresponding to the space-time signal, and unifying the timestamps of the plurality of pixel groups to obtain timestamp packets of the plurality of pixel groups; encoding based on the macro address of the plurality of pixel groups to obtain address packets of the plurality of pixel groups; and, for each pixel group, encoding based on the group address of the pixel group and the space-time signal of each pixel in the pixel group to obtain a space-time packet of each pixel group.

[0072] Three elements are essential for encoding: time, address, and signal. In related art, each pixel employs asynchronous sampling and output, so encoding for each pixel includes the pixel's event time, pixel address, and corresponding event polarity (corresponding to a signal), which inevitably results in a large amount of time redundancy and address redundancy. To alleviate this problem, the embodiments of the present disclosure employ global unified encoding for time, allowing multiple pixels to share a single time encoding, thereby reducing the redundancy of time encoding. For addresses, a unified encoding is employed using macro addresses and group addresses, allowing the pixel address range to be determined through the macro address, and the group addresses can be further combined to subdivide addresses with smaller granularity, thereby meeting usage needs and simultaneously reducing the redundancy of address encoding in related art.

[0073] 8 is a conceptual diagram of a signal packet provided by the presently disclosed embodiment. Referring to FIG. 8, when the multiple pixels shown in FIG. 6 are grouped to obtain m*14 pixel groups, an operation procedure for encoding the signal packet is shown. Here, the signal packet includes a timestamp packet, an address packet, and a space-time packet.

[0074] As shown in FIG. 8, the timestamp packet is a time encoding for every pixel, a sampling time to characterize the spatiotemporal signal of multiple pixels.

[0075] For the first row of pixels, address packet 1 and space-time packets 1_1 to 1_14 are provided, for the second row of pixels, address packet 2 and space-time packets 2_1 to 2_14 are provided, and so on, until for the mth row of pixels, address packet m and space-time packets m_1 to m_14 are provided. Here, the address packet has a macro address field that is used to characterize the macro addresses of multiple pixel groups, and the space-time packet has a group address field and a space-time code field that characterize the group address and space-time signal encoding of each pixel group, respectively.

[0076] For example, if the address of the pixel in the i-th row and column is add(i) and the address of the pixel in the i-th row and j-th column is add(i,j), the value of the macro address field in address packet 1 is the array address of the pixel in the first row, i.e., add(1). The value of the group address field in space-time packet 1_1 is add(1,1)-add(1,11), which corresponds to the 1st pixel to the 12th pixel in the 1st row, and the space-time code field is characterized as corresponding to the space-time signal encoding of the 1st pixel to the 12th pixel in the 1st row; the value of the group address field in space-time packet 1_2 is add(1,13)-add(1,24), which corresponds to the 13th pixel to the 24th pixel in the 1st row, and the space-time code field is characterized as corresponding to the space-time signal encoding of the 13th pixel to the 24th pixel in the 1st row; and so on; the value of the group address field in space-time packet 1_14 is add(1,157)-add(1,160), which corresponds to the 157th pixel to the 160th pixel in the 1st row, and the space-time code field is characterized as corresponding to the space-time signal encoding of the 157th pixel to the 160th pixel in the 1st row.

[0077] The value of the macro address field in address packet 2 is the array address of the second row pixel, i.e., add(2), and the value of the group address field in space-time packet 2_1 is add(2,1)-add(2,11), which corresponds to the 1st pixel to the 12th pixel in the second row, and the space-time code field corresponds to the space-time signal encoding of the 1st pixel to the 12th pixel in the second row. The other address packets and space-time packets are similar to this, so their description will be omitted here.

[0078] In some preferred embodiments, for the pixel group shown in Figure 7, the signal packet similarly includes a timestamp packet, an address packet, and a space-time packet. Here, the timestamp packet is a packet obtained by uniformly time-encoding all pixels based on the sampling time. Here, the first to fortieth pixel groups corresponding to the first to third rows correspond to one address packet, and the value of the macro address field in this packet is the array address of the pixels in the first to third rows, i.e., add(1)-add(3). Taking the first pixel group in the first to third rows as an example, the value of the group address field in the corresponding space-time packet is add(1,1)-add(3,4), which corresponds to the first to fourth pixels in the first row, the first to fourth pixels in the second row, and the first to fourth pixels in the third row. The space-time code field corresponds to the space-time signal encoding of the above 12 pixels.

[0079] In some preferred embodiments, the space-time signal includes a time dimension change amount and a space dimension difference amount, the time dimension change amount having a first preset data precision and the space dimension difference amount having a second preset data precision, and the space-time packet includes a time dimension sub-data packet and a space dimension sub-data packet, and the step of obtaining a space-time packet for each pixel group by encoding based on the group address of the pixel group and the space-time signal of each pixel in the pixel group includes the steps of: encoding the time dimension change amount of each pixel in the pixel group based on the first preset data precision to obtain a time dimension encoding and encoding the group address of the pixel group to obtain a first group address encoding; obtaining a time dimension sub-data packet for the pixel group based on the time dimension encoding and the first group address encoding; encoding the space dimension difference amount of each pixel in the pixel group based on the second preset data precision to obtain a space dimension encoding and encoding the group address of the pixel group to obtain a second group address encoding; and obtaining the space dimension sub-data packet for the pixel group based on the space dimension encoding and the second group address encoding. This processing method allows packets with different data precisions to be output using a unified output protocol, simplifying the output complexity and the subsequent decoding process.

[0080] In some preferred embodiments, the timestamp packet may further include a data precision field corresponding to a frame data precision ID (for example, the frame data precision ID can be determined based on the sampling period) in addition to the timestamp field. The address packet may further include a row data precision field corresponding to a row data precision ID. Alternatively, some extra data bits may be set as substitute bits in advance so that the substitute bits can be used to expand the field when it is necessary to expand it.

[0081] In some preferred embodiments, when considering some pixels in two adjacent sampling periods, the light intensity may not change or may change only slightly, so there may be some pixel groups in which the time dimension change amount of each pixel is small and the spatial dimension difference amount is almost the same as that of the previous sampling period. In such cases, there is no need to encode these pixel groups, thereby reducing the amount of encoding and data transfer.

[0082] In some preferred embodiments, pixel groups in the above situation can be screened out by setting a pixel group state. For example, a preset threshold is set in advance, and if the light intensity changes of each pixel in the pixel group between adjacent sampling periods are equal to or less than the preset threshold, the state of the pixel group is determined to be an invalid state; if the light intensity changes of at least one pixel in the pixel group between adjacent sampling periods are greater than the preset threshold, the state of the pixel group is determined to be a valid state.

[0083] In some preferred embodiments, the address packet includes address coding and valid pixel group coding, wherein the address coding is obtained by coding based on macro addresses of multiple pixel groups (corresponding to the macro address field), and the valid pixel group coding is obtained by coding based on the state of each pixel group, the state of the pixel group including a valid state and a disabled state, and the change in light intensity of each pixel in the disabled state pixel group between adjacent sampling periods is less than or equal to a preset threshold, and the change in light intensity of at least one pixel in the valid state pixel group between adjacent sampling periods is greater than the preset threshold.

[0084] Furthermore, for pixel groups in an invalid state, the amount of coding can be reduced by not encoding the space-time signal, and the space-time packets of such pixel groups can be discarded to reduce the amount of data transfer.

[0085] In addition, in some preferred embodiments, since the number of pixels in which light intensity changes occur between adjacent sampling periods is relatively small, the pixel time-dimension variations may contain a large number of zero values ​​or relatively small values, and the time-dimension variations have a certain sparsity. In view of this characteristic, in some preferred embodiments, the time-dimension variations may be encoded using a compression encoding method, thereby further reducing the amount of storage space and data transmission required.

[0086] In some preferred embodiments, the time dimension change amounts of the plurality of pixels in the pixel group correspond to a first matrix, the first matrix corresponding to a row vector format, a column vector format, or a matrix format. The step of encoding the time dimension change amount of each pixel in the pixel group based on a first preset data precision to obtain a time dimension encoding includes the steps of: generating a first flag bit matrix of the same size as the first matrix based on a result of comparing the time dimension change amount of each pixel in the pixel group with a first preset invalid value; generating a first compressed vector based on whether a value in the first matrix is ​​not an element of the first preset invalid value; and encoding the first flag bit matrix and the first compressed vector to obtain a time dimension encoding of the pixel group. The elements of the first flag bit matrix correspond to the elements in the first matrix. The elements in the first flag bit matrix include a first valid flag bit and a first invalid flag bit. The first valid flag bit characterizes that the time dimension change amount of the pixel corresponding to the element in the first matrix is ​​not the first preset invalid value, and the first invalid flag bit characterizes that the time dimension change amount of the pixel corresponding to the element in the first matrix is ​​the first preset invalid value. Here, the first preset invalid value may be zero or a small numerical value (for example, a time dimension change amount less than a preset threshold value can be regarded as the first preset invalid value).

[0087] For example, a pixel group corresponds to a pixel bar consisting of multiple adjacent pixel groups in a row, so the time dimension change amount of multiple pixels in a pixel group corresponds to a row vector format. Assuming that the size of the first matrix W1 is 1*12 and W1={12,0,0,23,4,56,0,0,16,0,0,0}, the first preset invalid value is 0, and the compression encoding process based on the first matrix is ​​as follows:

[0088] First, based on W1, a first flag bit matrix W2 with a size of 1*12 is generated, and W2={1,0,0,1,1,1,0,0,1,0,0,0}, the elements in W2 correspond one-to-one to the elements in W1, when the value of W1(1,j) is 0 (j represents the column number of the element), W2(1,j) corresponds to the first invalid flag bit and its value is 0, when W1(1,j) is a non-zero value, W2(1,j) corresponds to the first valid flag bit and its value is 1.

[0089] Next, columns in W1 that have all values ​​of 0 are deleted to obtain the first compressed vector W1', W1'={12, 23, 4, 56, 16}.

[0090] Finally, W1' and W2 are coded to obtain the temporal dimension coding of the pixel group.

[0091] For example, a pixel group corresponds to a pixel block consisting of multiple adjacent pixel groups in multiple rows, so the time dimension change amount of multiple pixels in a pixel group corresponds to a matrix format. Assuming that the size of the first matrix iW1 is 3*4 and W1={12,0,0,23;4,56,0,0;16,0,0,0}, the first preset invalid value is 0, and the compression encoding process based on the first matrix is ​​as follows:

[0092] First, based on W1, generate a first flag bit matrix W2 with a size of 3*4, where W2={1,0,0,1;1,1,0,0;1,0,0,0}, and the elements in W2 correspond one-to-one to the elements in W1. When W1(i,j) has a value of 0 (i and j represent the row number and column number of the element respectively), W2(i,j) corresponds to the first invalid flag bit and its value is 0; when W1(i,j) has a non-zero value, W2(i,j) corresponds to the first valid flag bit and its value is 1.

[0093] Next, all elements in W1 that have a value of 0 are deleted, and the remaining non-zero elements are rearranged to generate a first compressed vector W1', W1'={12, 23, 4, 56, 16}.

[0094] Finally, W1' and W2 are coded to obtain the temporal dimension coding of the pixel group.

[0095] Similarly, the spatial dimension difference amount can also be processed by adopting a compression encoding method.

[0096] In some preferred embodiments, the spatial dimension difference amounts of the plurality of pixels in the pixel group correspond to a second matrix, and the second matrix corresponds to a row vector format, a column vector format, or a matrix format. The step of encoding the spatial dimension difference amounts of each pixel in the pixel group based on the second preset data precision to obtain a spatial dimension encoding includes: generating a second flag bit matrix having the same size as the second matrix based on a result of comparing the spatial dimension difference amounts of each pixel in the pixel group with a second preset invalid value; generating a second compressed vector based on whether the value in the second matrix is ​​not an element of the second preset invalid value; and encoding the second flag bit matrix and the second compressed vector to obtain a spatial dimension encoding of the pixel group, wherein elements of the second flag bit matrix have a corresponding relationship with elements in the second matrix, and the elements in the second flag bit matrix include a second valid flag bit and a second invalid flag bit. The second valid flag bit characterizes that the spatial dimension difference amounts of the pixels corresponding to the elements in the second matrix are not the second preset invalid value, and the second invalid flag bit characterizes that the spatial dimension difference amounts of the pixels corresponding to the elements in the second matrix are the second preset invalid value. For details of the encoding process, please refer to the process of processing the time dimension variation amount, and the description will not be expanded here.

[0097] In some preferred embodiments, after obtaining the signal packet, the signal packet can be output to the outside, and the processor / processing core that receives the signal packet can perform corresponding processing (for example, decoding processing on the signal packet, imaging processing based on the decoded data, etc., embodiments of the present disclosure are not limited to this).

[0098] In light of the above, the embodiments of the present disclosure first adopt a synchronous sampling and output method to avoid the low compatibility issues caused by asynchronous processing methods and to be compatible with the spatiotemporal signal processing method of dual-mode image sensors. Second, pixel grouping can reduce the redundancy of address data. Furthermore, by fully utilizing the data sparsity, a compression encoding method can be adopted to compress the spatiotemporal signal, further reducing the amount of data, eliminating invalid information caused by a large number of zero values, and reducing bandwidth demands.

[0099] It should be understood that the above method embodiments mentioned in this disclosure can be combined with each other to form a combined embodiment as long as it does not violate the principle logic, and due to space limitations, this disclosure will not further explain it. Those skilled in the art can understand that in the above method of a specific embodiment, the specific execution order of each step should be determined by its function and possible inherent logic.

[0100] In a second aspect, embodiments of the present disclosure provide a data processing device based on an image sensor.

[0101] FIG. 9 is a block diagram of a data processing device based on an image sensor provided by an embodiment of the present disclosure.

[0102] Referring to FIG. 9, an embodiment of the present disclosure provides a data processing device based on an image sensor, and the data processing device 900 includes:

[0103] The acquisition module 910 is used for acquiring spatiotemporal signals of multiple pixels through an image sensor.

[0104] The grouping module 920 is used to group a plurality of pixels to obtain a plurality of pixel groups.

[0105] The determination module 930 is used to determine a group address of each pixel group and macro addresses of multiple pixel groups based on the pixel addresses within each pixel group.

[0106] The encoding module 940 is used to encode the signal packets of the plurality of pixel groups according to the sampling period corresponding to the spatio-temporal signal, the group address of each pixel group, the macro address of the plurality of pixel groups, and the spatio-temporal signal of the pixels in each pixel group to obtain signal packets of the plurality of pixel groups.

[0107] The embodiment provided by the present disclosure first acquires spatio-temporal signals of multiple pixels through an image sensor via an acquisition module. The spatio-temporal signals can determine not only the signal change status of the pixel itself in the time dimension, but also the signal difference status between the current pixel and its neighboring pixels, thereby more comprehensively reflecting the signal information of the pixel. Next, the multiple pixels are grouped to obtain multiple pixel groups. Taking into full consideration the locality of the spatio-temporal signal changes of the pixels, multiple pixels with relatively similar changes can be grouped into one pixel group through grouping, thereby reducing the additional redundancy of encoding address information for each change event during subsequent encoding. Furthermore, a determination module is used to determine group addresses for each pixel group and macro addresses for the multiple pixel groups based on the pixel addresses within each pixel group. During subsequent encoding, the macro addresses and group addresses can be coded with a higher granularity than coding each pixel address, thereby reducing the redundancy of address coding. Finally, encoding is performed based on the sampling period corresponding to the spatio-temporal signal, the group address of each pixel group, the macro address of the plurality of pixel groups, and the spatio-temporal signal of the pixels in each pixel group to obtain signal packets of the plurality of pixel groups, thereby reducing the time redundancy and address redundancy of the encoding and at the same time preserving the signal information of each pixel, thereby recovering the corresponding image based on the signal packets, and performing various tasks based on the image.

[0108] In a third aspect, embodiments of the present disclosure provide an image processing system.

[0109] FIG. 10 is a block diagram of an image processing system provided by an embodiment of the present disclosure.

[0110] Referring to FIG. 10 , an embodiment of the present disclosure provides an image processing system 1000, which includes an image sensor-based data processing device 1010 and at least one image sensor 1020, wherein: The image sensor 1020 is used to acquire spatiotemporal signals of a plurality of pixels based on a preset sampling period; The image sensor based data processing device 1010 is used to perform the image sensor based data processing method of any of the embodiments of the present disclosure.

[0111] The embodiment provided by the present disclosure collects and outputs spatiotemporal signals of pixels through an image sensor, groups the pixels, and during encoding, performs global encoding on the time of the spatiotemporal signals, and unifies the address encoding based on the macro address and group address of the pixel group, thereby alleviating the redundancy of time encoding and address encoding, and at the same time retaining the signal information of each pixel, thereby recovering the corresponding image based on the signal packet, and performing various tasks based on the image.

[0112] In addition, the present disclosure further provides an electronic device and a computer-readable storage medium, all of which can be used to realize any image sensor-based data processing method provided by the present disclosure. For corresponding technical solutions and descriptions, please refer to the corresponding descriptions in the method section, which will be omitted here.

[0113] FIG. 11 is a block diagram of an electronic device provided by an embodiment of the present disclosure.

[0114] Referring to FIG. 11 , an embodiment of the present disclosure provides an electronic device, the electronic device including at least one processor 1101, at least one memory 1102, and one or more I / O interfaces 1103 connected between the processor 1101 and the memory 1102, wherein the memory 1102 stores one or more computer programs executable by the at least one processor 1101, and the one or more computer programs are executed by the at least one processor 1101 so as to enable the at least one processor 1101 to perform the above-mentioned image sensor-based data processing method; FIG. 12 is a block diagram of an electronic device provided by an embodiment of the present disclosure.

[0115] Referring to FIG. 12, an embodiment of the present disclosure provides an electronic device, which includes a plurality of processing cores 1201 and an on-chip network 1202, where the plurality of processing cores 1201 are all connected to the on-chip network 1202, and the on-chip network 1202 is used for interaction of data between the plurality of processing cores and external data.

[0116] Here, one or more processing cores 1201 store one or more instructions to be executed by the one or more processing cores 1201 so that the one or more processing cores 1201 can perform the above-mentioned image sensor-based data processing method.

[0117] In some embodiments, the electronic device may be a neural network chip, which may employ vectorized computation and require access to parameters such as neural network model weight information via external memory, such as a double data rate (DDR) synchronous dynamic random memory. Thus, embodiments of the present disclosure employ batch processing for high computational efficiency.

[0118] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor / processing core, realizes the image sensor-based data processing method described above. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0119] An embodiment of the present disclosure further provides a computer program product, comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, which, when executed by a processor of an electronic device, causes the processor in the electronic device to perform the image sensor-based data processing method described above.

[0120] Those skilled in the art will understand that all or some of the steps of the methods, systems, and functional modules / units of the devices disclosed above can be implemented as software, firmware, hardware, or any suitable combination thereof. In hardware embodiments, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components. For example, one physical component may have multiple functions, or one function or step may be performed by multiple physical components working together. Some or all of the physical components can be implemented as software executed by a processor, such as a central processor, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as a dedicated integrated circuit. Such software can be distributed on a computer-readable storage medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium).

[0121] As known to those skilled in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information (e.g., computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compressed disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium used to store the desired information and accessible by a computer. Furthermore, it is known to those skilled in the art that communication media generally include computer-readable program instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery media.

[0122] The computer-readable program instructions described herein can be downloaded to each computing / processing device from a computer-readable storage medium, or can be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium stored in each computing / processing device.

[0123] Computer program instructions for carrying out the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or target code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" language, or similar programming languages. The computer-readable program instructions may execute entirely on the user computer, partially on the user computer, as a separate software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. When referring to a remote computer, the remote computer may be connected to the user computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., connected via the Internet using an Internet Service Provider). In some embodiments, the state information of the computer-readable program instructions can be used to customize custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), that can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0124] The computer program products described herein may be tangibly embodied in hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is tangibly embodied as a computer storage medium, and in another alternative embodiment, the computer program product is tangibly embodied as a software product, such as a software development package (SOftWaRe DevelOpment Kit, SDK).

[0125] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0126] These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus such that, when executed by the processor of the computer or other programmable data processing apparatus, it can produce a machine that generates an apparatus that implements the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can be stored on a computer-readable storage medium to cause a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable medium having the instructions stored thereon comprises an article of manufacture, the article of manufacture containing instructions that implement various aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0127] The computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device and cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to realize a computer-implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0128] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment, or part of an instruction, which includes one or more executable instructions for implementing a given logical function. In some alternative implementations, the functions displayed in the blocks may occur in a different order than the order displayed in the figures. For example, two consecutive blocks may actually be executed essentially in parallel, or they may be executed in the reverse order depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented in a dedicated hardware-based system that performs the given functions or operations, or in a combination of dedicated hardware and computer instructions.

[0129] Although exemplary embodiments have been disclosed and specific terms have been used herein, these are generally used and should be construed in an illustrative sense only and not as limiting. It will be apparent to those skilled in the art that, in some embodiments, features, characteristics, and / or elements described in connection with a particular embodiment may be used alone, unless otherwise specified, or may be used in combination with features, characteristics, and / or elements described in connection with other embodiments. Accordingly, those skilled in the art will recognize that various modifications in form and detail may be made without departing from the scope of the present disclosure, as defined by the appended claims.

Claims

1. acquiring spatiotemporal signals of a plurality of pixels via an image sensor; grouping a plurality of pixels to obtain a plurality of pixel groups; determining a group address for each of the pixel groups and a macro address for a plurality of the pixel groups based on pixel addresses within each of the pixel groups; Encoding the signal packets of the plurality of pixel groups according to a sampling period corresponding to the spatio-temporal signal, a group address of each of the pixel groups, a macro address of the plurality of pixel groups, and the spatio-temporal signal of the pixels in each of the pixel groups; A data processing method based on an image sensor, comprising:

2. The step of acquiring spatiotemporal signals of a plurality of pixels via an image sensor includes: acquiring spatiotemporal signals of a plurality of pixels via the image sensor based on a preset sampling period, the spatiotemporal signals being used to characterize at least signal change information of the pixels in a time dimension and signal difference information in a space dimension; 2. The method of claim 1, comprising:

3. The step of acquiring spatiotemporal signals of a plurality of pixels via the image sensor based on a preset sampling period includes: sampling output data of the image sensor based on the sampling period to obtain spatiotemporal signals of a plurality of pixels corresponding to the image sensor; Or, When the image sensor outputs data to the outside based on the sampling period, obtaining spatiotemporal signals of corresponding pixels based on output data of the image sensor; 3. The method of claim 2, comprising:

4. the spatiotemporal signal is a light intensity signal, and the spatiotemporal signal includes at least a time dimension change amount and a space dimension difference amount; and the image sensor is further used to acquire color signals of a plurality of pixels; the time dimension change amount is a change amount between the light intensity of the pixel in a current sampling period and the light intensity of the pixel in a previous sampling period, and the space dimension difference amount is a difference amount between the light intensity of the pixel in a current sampling period and the light intensity of at least one adjacent pixel in a current sampling period; The method of claim 1.

5. the time dimension change amount has a first preset data precision, and the space dimension difference amount has a second preset data precision; The first preset data precision corresponds to the same data precision as the second preset data precision, or the first preset data precision corresponds to a data precision different from the second preset data precision; The method of claim 4.

6. The step of grouping a plurality of pixels to obtain a plurality of pixel groups comprises: Dividing a pixel bar consisting of a plurality of adjacent pixel groups in one array into one pixel group to obtain a plurality of said pixel groups; and / or Dividing a pixel block consisting of a plurality of adjacent pixel groups in the plurality of arrays into one pixel group to obtain a plurality of said pixel groups; The method of claim 1.

7. The step of grouping a plurality of pixels to obtain a plurality of pixel groups comprises: determining the number of pixels in each pixel group based on the data precision of the spatio-temporal signal and the format requirements of the signal packet; determining a grouping size based on the pixel data in each pixel group, the grouping size being used to characterize the data size of the pixels in the pixel group and corresponding to a vector format and / or a matrix format; grouping a plurality of the pixels based on the grouping size to obtain a plurality of the pixel groups; The method of claim 1.

8. The signal packets include a timestamp packet, an address packet, and a space-time packet; the step of obtaining signal packets of the plurality of pixel groups by encoding based on a sampling period corresponding to the spatio-temporal signal, a group address of each of the pixel groups, a macro address of a plurality of the pixel groups, and the spatio-temporal signal of pixels in each of the pixel groups, determining timestamps for the plurality of pixel groups based on a sampling period corresponding to the spatio-temporal signal, and unified-encoding the timestamps for the plurality of pixel groups to obtain timestamp packets for the plurality of pixel groups; encoding based on macro addresses of a plurality of pixel groups to obtain address packets of the plurality of pixel groups; for each of the pixel groups, encoding the pixel groups based on a group address of the pixel group and the spatio-temporal signals of each pixel in the pixel group to obtain a spatio-temporal packet of each of the pixel groups; The method of claim 1.

9. the address packet includes an address encoding and a valid pixel group encoding; the address coding is obtained by coding based on macro addresses of a plurality of the pixel groups, and the valid pixel group coding is obtained by coding based on the states of each of the pixel groups, the states of the pixel groups including a valid state and an invalid state, and the amount of change in light intensity of each pixel in an invalid state between adjacent sampling periods is equal to or less than a preset threshold, and the amount of change in light intensity of at least one pixel in an valid state between adjacent sampling periods is greater than the preset threshold; The method of claim 8.

10. After the step of encoding based on the group address of the pixel group and the spatio-temporal signal of each pixel in the pixel group to obtain a spatio-temporal packet of each pixel group, If the state of the pixel group is an invalid state, discarding the space-time packet of the pixel group. The method of claim 9.

11. the space-time signal includes a time dimension change amount and a space dimension difference amount, the time dimension change amount has a first preset data precision, the space dimension difference amount has a second preset data precision, the space-time packet includes a time dimension sub-data packet and a space dimension sub-data packet; The step of obtaining a space-time packet for each pixel group by encoding based on a group address of the pixel group and the space-time signal of each pixel in the pixel group includes: encoding a time dimension change amount of each pixel in the pixel group based on the first preset data precision to obtain a time dimension encoding, and encoding a group address of the pixel group to obtain a first group address encoding; obtaining a time dimension sub-data packet of the pixel group based on the time dimension encoding and the first group address encoding; encoding a spatial dimension difference amount of each pixel in the pixel group based on the second preset data precision to obtain a spatial dimension encoding, and encoding a group address of the pixel group to obtain a second group address encoding; and obtaining a spatial dimension sub-data packet of the pixel group based on the spatial dimension encoding and the second group address encoding. The method of claim 8.

12. a time dimension change amount of a plurality of pixels in the pixel group corresponds to a first matrix, and the first matrix corresponds to a row vector format, a column vector format, or a matrix format; The step of encoding the time dimension change amount of each pixel in the pixel group based on the first preset data precision to obtain time dimension encoding includes: generating a first flag bit matrix having the same size as the first matrix based on a comparison result between the time dimension change amount of each pixel in the pixel group and a first preset invalid value; generating a first compressed vector based on the value in the first matrix being not an element of the first preset invalid value; encoding the first flag bit matrix and the first compressed vector to obtain a time dimension encoding of the pixel group; The elements of the first flag bit matrix have a corresponding relationship with the elements in the first matrix, and the elements in the first flag bit matrix include a first valid flag bit and a first invalid flag bit, the first valid flag bit characterizing that the time dimension change amount of the pixel corresponding to the element in the first matrix is ​​not the first preset invalid value, and the first invalid flag bit characterizing that the time dimension change amount of the pixel corresponding to the element in the first matrix is ​​the first preset invalid value. The method of claim 11.

13. The spatial dimension difference amount of the plurality of pixels in the pixel group corresponds to a second matrix, and the second matrix corresponds to a row vector format, a column vector format, or a matrix format; The step of encoding the spatial dimension difference amount of each pixel in the pixel group based on the second preset data precision to obtain spatial dimension encoding includes: generating a second flag bit matrix having the same size as the second matrix based on a comparison result between the spatial dimension difference amount of each pixel in the pixel group and a second preset invalid value; generating a second compressed vector based on the value in the second matrix being not an element of the second preset invalid value; encoding the second flag bit matrix and the second compressed vector to obtain a spatial dimension encoding of the pixel group; The elements of the second flag bit matrix have a corresponding relationship with the elements in the second matrix, and the elements in the second flag bit matrix include a second valid flag bit and a second invalid flag bit, the second valid flag bit characterizing that the spatial dimension difference amount of the pixel corresponding to the element in the second matrix is ​​not the second preset invalid value, and the second invalid flag bit characterizing that the spatial dimension difference amount of the pixel corresponding to the element in the second matrix is ​​the second preset invalid value. The method of claim 11.

14. an acquisition module for acquiring spatiotemporal signals of a plurality of pixels via an image sensor; a grouping module for grouping the plurality of pixels to obtain a plurality of pixel groups; a determination module for determining a group address for each of the pixel groups and a macro address for a plurality of the pixel groups based on pixel addresses within each of the pixel groups; an encoding module for encoding based on a sampling period corresponding to the spatio-temporal signal, a group address of each of the pixel groups, a macro address of a plurality of the pixel groups, and the spatio-temporal signal of pixels in each of the pixel groups to obtain signal packets of the plurality of pixel groups; Data processing device based on image sensor.

15. an image sensor-based data processing device; and at least one image sensor; The image sensor is used to acquire spatiotemporal signals of a plurality of pixels based on a preset sampling period; The image sensor based data processing device is used to perform the image sensor based data processing method according to any one of claims 1 to 13. Image processing system.

16. at least one processor; a memory communicatively coupled to the at least one processor; the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor such that the at least one processor is capable of performing the image sensor based data processing method of any one of claims 1 to 13. Electronic devices.

17. a computer program stored therein which, when executed by a processor, is capable of carrying out the image sensor-based data processing method according to any one of claims 1 to 13; A computer-readable storage medium.

18. computer-readable code or a non-volatile computer-readable storage medium carrying said computer-readable code, which, when executed by a processor of an electronic device, causes the processor in said electronic device to perform the image sensor-based data processing method of any one of claims 1 to 13. Computer program products.

Images