Image generation method and device, electronic equipment and storage medium

Abstract

The invention discloses an image generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring an event stream within a preset time interval, wherein an event in the event stream comprises a coordinate generated by the event and a timestamp; dividing a preset time interval to obtain at least two time periods; for each time period, counting all target events in the time period, determining target coordinates for generating the target events and the number of the target events generated by each target coordinate, determining a pixel value of each target coordinate according to the number of the target events generated by each target coordinate, and forming an initial image corresponding to the time period; the target image is determined according to the initial image corresponding to each time period, so that the problems of relatively low speed and relatively low accuracy when the image is generated according to the event stream data are solved, the conversion process is relatively simple, and information loss is avoided; motion blur generated when an object moves too fast can be reduced, and the image detection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an image generation method, device, electronic equipment and storage medium. Background Art

[0002] In the prior art, image acquisition is usually performed through an ordinary camera. This method requires a lot of redundant data to be processed, has high requirements on the processor, and has a low dynamic range rate. In order to improve data processing efficiency, event cameras can be used to collect data, which can greatly reduce redundant data, thereby improving the computational efficiency of the post-processing algorithm. However, event cameras do not have the concept of "frames". When changes occur in the real scene, the event camera will generate some pixel-level outputs (i.e., events). An event specifically includes (t, x, y, p), where x and y are the pixel coordinates of the event in 2D space, t is the timestamp of the event, and p is the polarity of the event (if the brightness is enhanced, p = +1 indicates a positive event, otherwise, P = -1 indicates a negative event). When using event camera data, it is usually necessary to convert it into an image. The existing conversion method will perform statistics and denoising on the event stream data of the event camera over a period of time, and generate images in a gradient manner. The process is relatively cumbersome and it is easy to filter out effective information. Therefore, how to quickly and accurately generate images based on event stream data has become a problem to be solved. Summary of the invention

[0003] The present invention provides an image generation method, device, electronic device and storage medium to solve the problem of slow speed and low accuracy when generating images according to event stream data.

[0004] According to one aspect of the present invention, there is provided an image generation method, comprising:

[0005] Acquire an event stream within a preset time interval, wherein the events in the event stream include coordinates and timestamps of event generation;

[0006] Dividing the preset time interval into at least two time periods;

[0007] For each time period, all target events within the time period are counted, the target coordinates generating the target events and the number of target events generated by each target coordinate are determined, the pixel value of each target coordinate is determined according to the number of target events generated by each target coordinate, and an initial image corresponding to the time period is formed;

[0008] The target image is determined according to the initial images corresponding to the time periods.

[0009] According to another aspect of the present invention, there is provided an image generating device, comprising:

[0010] An event stream acquisition module, used to acquire an event stream within a preset time interval, wherein an event in the event stream includes a coordinate and a timestamp of the event;

[0011] A time period division module, used for dividing the preset time interval into at least two time periods;

[0012] An initial image generation module is used to count all target events in each time period, determine the target coordinates that generate the target events and the number of target events generated by each target coordinate, determine the pixel value of each target coordinate according to the number of target events generated by each target coordinate, and form an initial image corresponding to the time period;

[0013] The target image determination module is used to determine the target image according to the initial image corresponding to each of the time periods.

[0014] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image generating method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image generation method described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention is to obtain an event stream within a preset time interval, wherein the events in the event stream include the coordinates and timestamps of event generation; divide the preset time interval into at least two time periods; for each time period, count all target events within the time period, determine the target coordinates that generate the target event and the number of target events generated by each target coordinate, determine the pixel value of each target coordinate according to the number of target events generated by each target coordinate, and form an initial image corresponding to the time period; determine the target image according to the initial image corresponding to each time period, thereby solving the problem of slow speed and low accuracy when generating images based on event stream data, and collecting the event stream within the preset time interval. In order to avoid the problem of high data volume, the preset time interval is divided to obtain at least two time periods. For each time period, the target coordinates of the target events generated in the time period are counted, and the number of all target events generated by the target coordinates is counted. The pixel values ​​of the target coordinates are assigned according to the number of target events to form an initial image corresponding to this time period. The target image is formed based on the initial images corresponding to each time period to complete the conversion of the event stream to the image. The conversion process is relatively simple, and all valid information can be effectively retained to avoid information loss, thereby improving the conversion speed and accuracy of the event stream data to the image. Dividing the data into at least two sections for processing can reduce the motion blur caused by the object moving too fast, thereby further improving the image detection efficiency.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flowchart of an image generating method provided according to Embodiment 1 of the present invention;

[0023] Figure 2 is a flowchart of an image generating method provided according to Embodiment 2 of the present invention;

[0024] Figure 3 is a network structure diagram of a deep learning model provided according to Embodiment 2 of the present invention;

[0025] Figure 4is a network structure diagram of a convolution block provided according to Embodiment 2 of the present invention;

[0026] Figure 5 is a structural schematic diagram of an image generating device provided according to Embodiment 3 of the present invention;

[0027] Figure 6 It is a schematic diagram of the structure of an electronic device for implementing the image generating method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 This is a flowchart of an image generation method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where an image is generated according to an event stream. The method can be executed by an image generation device. The image generation device can be implemented in the form of hardware and / or software. The image generation device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S101. Acquire an event stream within a preset time interval, where an event in the event stream includes a coordinate and a timestamp of event generation.

[0033] In this embodiment, the preset time interval can be specifically understood as a pre-set time interval, and the size of the preset time interval can be set according to the requirements for data processing accuracy in the actual application process; if the preset time interval is too large, it will cause data redundancy and will also affect the accuracy of the data; if the preset time interval is too small, it will cause insufficient accuracy due to the small amount of data, for example, 30ms, 100ms, etc.

[0034] The event stream is collected by an event camera, which can be installed on a movable device or set in a fixed position. The event camera is a sensor that "only senses moving objects". There is an independent photoelectric sensor module at each pixel of the event camera. When the brightness change at the pixel exceeds the set threshold, event data (sometimes also called pulse data) will be generated and output; because all pixels work independently, the data output of the event camera is asynchronous and sparse in space. The event camera collects the event stream in real time and records the events that have changed. The preset time interval is set in advance, and the event stream within the preset time interval is obtained. The event stream within the preset time interval is used to determine a frame of image. The event stream includes events that record changes. The event includes coordinates and timestamps. The coordinates are used to indicate the location where the event occurred, and the timestamp is used to indicate the time when the event occurred.

[0035] It should be noted that the event may also include polarity, which is used to indicate whether the brightness of the event increases or decreases, and the polarity may be positive or negative. In the embodiment of the present application, polarity is ignored when converting the event stream into an image, which can reduce the data processing intensity.

[0036] S102: Divide the preset time interval into at least two time periods.

[0037] In the embodiment of the present application, when dividing the preset time interval, the division may be non-uniform or equal. For example, the equal division is N parts, and N time periods are obtained, where N is greater than or equal to 2, and the length of each time period is related to the preset time interval; or, the division is performed according to a certain preset duration, and the preset duration is less than the preset time interval, and the preset time interval is divided according to the preset duration to obtain M time periods. If the remaining duration is less than the preset duration, the remaining duration is directly used as the last time period. When the division is performed in this way, the duration of the last time period may be less than the previous time period.

[0038] S103, for each time period, counting all target events within the time period, determining the target coordinates that generate the target events and the number of target events generated by each target coordinate, determining the pixel value of each target coordinate according to the number of target events generated by each target coordinate, and forming an initial image corresponding to the time period.

[0039] In this embodiment, the target event can be specifically understood as an event divided into a specific time period; the target coordinates can be specifically understood as the coordinates that generate the target event; and the initial image can be specifically understood as an image generated based on event data within a certain time period.

[0040] For each time period, a corresponding initial image can be generated according to the target events contained in this time period. The initial image is generated as follows: determine the time range corresponding to the time period, filter all events in the event stream according to the time range, filter out events with timestamps within this time range, and determine these events as target events generated within this time period. Determine the target coordinates corresponding to each target event according to the coordinates recorded in the event. For each coordinate, it can generate events at different time points. Therefore, the number of target events generated by each target coordinate is not unique. Count the number of all target events generated by the target coordinates within this time period. Analyze the number of target events corresponding to each target coordinate, and perform calculations based on the number of target events generated by each target coordinate. For example, determine the number of target events generated by the target coordinate as the pixel value of the target coordinate, or scale, normalize, etc. the number of target events, and use the obtained value as the pixel value of the target coordinate. For coordinates where there is no event, the pixel value can be fixedly assigned, for example, assigned to 0 or 255, and the initial image is formed based on the obtained pixel value of each target coordinate and the pixel value of the coordinate where there is no target event.

[0041] S104: Determine the target image according to the initial images corresponding to each time period.

[0042] In this embodiment, the target image can be specifically understood as an image finally formed according to the event stream, and the target image is formed based on the event stream within a preset time interval.

[0043] After obtaining the initial images corresponding to each time period, each initial image is processed, for example, each initial image is spliced, pixel averages are calculated, pixel values ​​are weighted summed, etc., to obtain a target image.

[0044] The image generation method provided by the embodiment of the present invention solves the problems of slow speed and low accuracy when generating images based on event stream data. By collecting the event stream within a preset time interval, the problem of high data volume can be avoided, and the preset time interval is divided to obtain at least two time periods. For each time period, the target coordinates of the target event generated within the time period are counted, and the number of all target events generated by the target coordinates is counted. The pixel values ​​of the target coordinates are assigned according to the number of target events to form an initial image corresponding to this time period. The target image is formed based on the initial images corresponding to each time period, and the conversion of the event stream to the image is completed. The conversion process is relatively simple, and all valid information can be effectively retained to avoid information loss, thereby improving the conversion speed and accuracy of the event stream data to the image. Dividing the data into at least two sections for processing can reduce the motion blur caused by the object moving too fast, and further improve the image detection efficiency.

[0045] Embodiment 2

[0046] Figure 2 This is a flow chart of an image generation method provided in Embodiment 2 of the present invention. This embodiment is refined on the basis of the above embodiment. Figure 2 As shown, the method includes:

[0047] S201. Acquire an event stream within a preset time interval, where an event in the event stream includes a coordinate and a timestamp of event generation.

[0048] S202: Divide the preset time interval according to a preset number to obtain a time period.

[0049] In this embodiment, the preset number can be pre-set and can remain unchanged during the image generation process; the preset number is an integer, the preset number is greater than or equal to 2, and the preset number is determined according to the size of the preset time interval.

[0050] When the object moves too fast, the resulting image will be blurred. If the data is divided into n parts, the motion blur will be relatively reduced; a suitable n value can improve the detection rate of the data. If n is too large, each initial image will not be able to map too much data, resulting in a lower recognition rate. Select the appropriate preset number according to the size of the preset time interval in advance. For example, when the selected time is 100ms, n can be set to 15; when the selected time is 30ms, n can be set to 5.

[0051] When generating images, the event stream can be collected in real time, and the corresponding target images can be continuously generated. During the collection process, the preset time interval remains unchanged, and the preset number also remains unchanged accordingly.

[0052] S203 , for each time period, counting all target events within the time period, determining the target coordinates generating the target event and the number of target events generated by each target coordinate.

[0053] S204: Count the number of target events generated by each target coordinate, and determine the maximum number and the minimum number.

[0054] Compare the number of target events generated by each target coordinate to determine the maximum and minimum numbers. For example, a sorting algorithm sorts the numbers to obtain the maximum and minimum numbers. The sorting algorithm may be bubble sort, selection sort, quick sort, etc.

[0055] S205 , calculating a first difference between the maximum number and the minimum number, and determining a ratio of a preset gray value to the first difference as a pixel coefficient.

[0056] In this embodiment, the first difference is the difference between the maximum number and the minimum data. The preset grayscale value can be 1 or 255. The preset grayscale value in the embodiment of the present application preferably adopts 255 to better represent the data information. The pixel coefficient can be specifically understood as the coefficient used when converting the number of events into pixel values.

[0057] A preset grayscale value is set in advance, and the difference obtained by subtracting the minimum number from the maximum number is the first difference. The ratio obtained by dividing the preset grayscale value by the first difference is used as the pixel coefficient.

[0058] S206 . For each target coordinate, calculate a second difference between the number of target events generated by the target coordinate and the minimum number, and determine the product of the second difference and the pixel coefficient as the pixel value of the target coordinate.

[0059] In this embodiment, the second difference is the difference between the number of target events and the minimum number.

[0060] For each target coordinate, its corresponding pixel value can be calculated in the following manner: subtract the minimum number from the number of target events generated by the target coordinate to obtain a second difference, calculate the product of the second difference and the pixel coefficient, and use the product as the pixel value of the target coordinate.

[0061] Exemplarily, the present application embodiment provides a formula for calculating the pixel value of the target coordinates:

[0062]

[0063] Among them, E ij is the pixel value of the target coordinate (i, j), p ijis the number of target events generated by the target coordinates (i, j), min(p) is the minimum number, max(p) is the maximum number, and 255 is the preset grayscale value.

[0064] S207 . Based on the pixel values ​​corresponding to the target coordinates, an initial image corresponding to the time period is formed.

[0065] The pixel values ​​are sorted according to the corresponding target coordinates, and the coordinates where no event exists are assigned a preset value, which may be 0, to form an initial image corresponding to this time period.

[0066] In the embodiment of the present application, the initial image corresponding to each time period is determined through steps S203 to S207.

[0067] S208 , stitching the initial images in sequence according to the coordinates to obtain a target image with a preset number of spatial dimensions.

[0068] After obtaining each initial image, each coordinate point in each initial image is stitched in sequence according to the one-to-one correspondence of the same coordinates to obtain a target image with a preset number of spatial dimensions, specifically a target image in a three-dimensional space, in which the coordinates constitute a two-dimensional plane, and the preset number constitutes the third dimension, namely the spatial dimension.

[0069] In the embodiment of the present application, the maximum number and the minimum number are determined by counting the number of target events, and then the pixel coefficient is determined; based on the pixel coefficient and the second difference between the number of target events and the minimum number, the pixel value of the target coordinate is assigned to determine the pixel value; the process is simple and convenient, and only needs to pay attention to whether the event has occurred, without paying attention to the polarity of the event, thereby reducing the amount of data processing. The size of the pixel value corresponding to each coordinate point in the initial image is positively correlated with the number of generated events. The more events are generated, the greater the brightness of the value on the pixel (the parameter for measuring color accuracy).

[0070] Optionally, the event stream includes eye features.

[0071] In this embodiment, the eye features can be specifically understood as feature data used to represent the eyes, for example, pupil information (such as pupil position, pupil shape), iris position, iris shape, eyelid position, eye corner position, light spot (also called Purkin spot) information (such as light spot position), etc. Specific to this embodiment, pupil information can be pupil center, ellipse major and minor axis lengths, and major axis angles. In order to facilitate the collection of eye features, the event camera can be installed on eye movement devices, AR / VR glasses and other devices, and the user can wear eye movement devices and other devices. When the user wears them, the event camera can collect the user's eye features. When the event stream contains eye features, the target image at this time can be used to locate the pupil, iris, eyelid or eye corner.

[0072] As an optional embodiment of this embodiment, this optional embodiment is further optimized on the basis that the event stream includes eye features, and the method further includes:

[0073] A1. Input the target image into the pre-trained deep learning model to obtain the center point feature map, offset feature map, and width and height feature map output by the deep learning model.

[0074] In this embodiment, the deep learning model is a network model that can be identified based on experience after training. The center point feature map can be specifically understood as a feature map containing the center point of the pupil; the offset feature map can be specifically understood as a feature map including the offset of the pupil; and the width and height feature map can be specifically understood as a feature map containing the width and height of the pupil.

[0075] Pre-train the deep learning model. To reduce the amount of data processing, a model with fewer convolution layers can be used. Obtain training samples, which include images to be trained formed according to event streams within a preset time interval and label values ​​of the images to be trained. Input the training samples into the deep learning model for training in sequence, and obtain the predicted value after processing by each convolution layer of the deep learning model. The predicted value is compared with the label value, and the loss function is calculated. The parameters of the deep learning model are adjusted by back propagation through the loss function until a deep learning model that meets the requirements is obtained. In the embodiment of the present application, the deep learning model can output three feature maps, which represent the center point, offset, and width and height respectively. Since the three feature maps represent different types of data respectively, compared with containing different types of data in one image at the same time, the model parameters can be accurately adjusted during the model training process, the model accuracy is higher, and the recognition results are more accurate and reliable. After the model is trained, the target image is input into the pre-trained deep learning model. The deep learning model recognizes the target image according to the experience learned during the training process, and obtains and outputs the center point feature map, the offset feature map, and the width and height feature map.

[0076] For example, Figure 3A network structure diagram of a deep learning model is provided, taking 2 convolution layers and 4 convolution blocks as an example, the convolution kernel of the first convolution layer 31 is 3*3, the step size S=2, the convolution kernel of the second convolution layer 36 is 3*3, the step size S=1, the convolution kernel of each convolution block is 3*3, the step size S=2, and the number of convolution layers is N. The target image 30 of n*h*w is input to the first convolution layer 31 for processing, wherein n is the dimension of the target image 30, n is equal to the preset number, h is the height of the target image, w is the width of the target image, and 1 is the dimension of the target image. The output result of the convolution layer 31 is input to the first convolution block 32, where N=2. The convolution block 32 performs convolution processing and inputs the output result to the convolution block 33, where N=3. The convolution block 33 performs convolution processing and inputs the output result to the convolution block 34, where N=3. The convolution block 34 performs convolution processing and inputs the output result to the convolution block 35, where N=2. The convolution block 35 performs convolution processing and multiplies the output result by 2. Sampling; adding the 2x upsampling result to the output result of the convolution block 34; upsampling the added result by 2x, adding the 2x upsampling result to the output result of the convolution block 33, repeatedly upsampling the added result by 2x, and then continuing to add the 2x upsampling result to the output result of the convolution block 32, and inputting the added result to the second convolution layer 36, the convolution layer 36 processes the input data and outputs a center point feature map 37, an offset feature map 38 and a width and height feature map 39.

[0077] For example, Figure 4 A network structure diagram of a convolution block is provided. The convolution block contains N layers, that is, the convolution block is composed of N convolution layers. The convolution kernel of each convolution layer is 3*3, the step size of the first convolution layer is S=2, and the step size of the convolution layers from the second layer to the Nth layer is S=1.

[0078] A2. Determine the pupil location information based on the center point feature map, offset feature map, and width and height feature map.

[0079] The center point feature map contains the position information of the center point of the pupil. The center point feature map is identified to determine the position information of the center point of the pupil. For example, the coordinates corresponding to the maximum pixel value in the center point feature map are determined as the coordinates of the center point of the pupil; or the center point feature map is processed, for example, by maximum pooling, mean pooling, etc., and the coordinates of the center point of the pupil are identified according to the processed results. The center point of the pupil is further offset corrected according to the offset feature map, and the width and height of the center point of the pupil are obtained from the width and height feature map according to the coordinates of the center point of the pupil, and the position information of the pupil is determined based on the coordinates, offset, width and height of the center point of the pupil.

[0080] As an optional embodiment of this embodiment, this optional embodiment further optimizes the pupil position information determined based on the center point feature map, the offset feature map and the width and height feature map as follows:

[0081] B1. Perform a maximum pooling operation on the center point feature map, and determine the coordinates of the maximum pixel value in the center point feature map after pooling as the coordinates of the pupil center point.

[0082] Perform a maximum pooling operation on the center point feature map. The area size of the maximum pooling operation can be set according to demand, for example, 3*3, that is, perform a maximum pooling operation on the center point feature map according to 3*3, and take the maximum value of the pixel value in the 3*3 area as the pixel value of this area, process each area in turn, obtain the center point feature map after pooling, compare the size of the pixel value of each coordinate point in the center point feature map after pooling, obtain the maximum pixel value, and determine the coordinates of the maximum pixel value as the coordinates of the center point of the pupil.

[0083] B2. Correct the coordinates of the pupil center point according to the offset feature map to obtain the target pupil center point coordinates.

[0084] In this embodiment, the target pupil center coordinates can be specifically understood as the corrected pupil center coordinates. According to the pupil center coordinates, the corresponding value in the offset feature map is determined as the offset, and the pupil center coordinates are corrected according to the offset to obtain the target pupil center coordinates.

[0085] As an optional embodiment of this embodiment, this optional embodiment further corrects the coordinates of the pupil center point according to the offset feature map to obtain the target pupil center point coordinates, which are optimized as follows:

[0086] B21. Determine the value of the position corresponding to the coordinates of the pupil center point in the first channel of the offset feature map as the horizontal coordinate offset, and determine the sum of the horizontal coordinate offset and the horizontal coordinate in the coordinates of the pupil center point as the corrected horizontal coordinate.

[0087] In this embodiment, the horizontal axis offset can be specifically understood as the offset of the horizontal axis; the first channel can be specifically understood as a channel in the offset feature map, and the offset feature map includes multiple channels to store offsets in different directions.

[0088] The coordinate axis directions corresponding to the values ​​stored in different channels in the offset feature map are predefined. The embodiment of the present application takes the offset of the horizontal coordinate stored in the first channel as an example. The value of the corresponding position in the first channel of the offset feature map is determined according to the coordinates of the pupil center point, and this value is determined as the horizontal coordinate offset. The horizontal coordinate in the coordinates of the pupil center point is determined, and the horizontal coordinate plus the horizontal coordinate offset is obtained. The sum is the corrected horizontal coordinate.

[0089] B22. Determine the value of the position corresponding to the coordinates of the pupil center point in the second channel of the offset feature map as the ordinate offset, and determine the sum of the ordinate offset and the ordinate in the coordinates of the pupil center point as the corrected ordinate.

[0090] In this embodiment, the vertical coordinate offset can be specifically understood as the offset of the vertical coordinate; the second channel can be specifically understood as another channel in the offset feature map, which stores the offset in the vertical axis direction.

[0091] According to the coordinates of the pupil center point, the value of the corresponding position in the second channel of the offset feature map is determined, and this value is determined as the ordinate offset. The ordinate in the coordinates of the pupil center point is determined, and the ordinate offset is added to the ordinate, and the sum obtained is the corrected ordinate.

[0092] B23. Determine the target pupil center coordinates based on the corrected horizontal coordinate and the corrected vertical coordinate.

[0093] The corrected abscissa and the corrected ordinate are used as the abscissa and ordinate in the target pupil center coordinates to obtain the target pupil center coordinates.

[0094] B3. The value of the position corresponding to the coordinates of the pupil center point in the first channel of the width-height feature map is determined as the pupil width, and the value of the position corresponding to the coordinates of the pupil center point in the second channel of the width-height feature map is determined as the pupil height.

[0095] In this embodiment, the width and height feature map also includes two channels, which respectively store the width and height of the pupil. For example, the first channel stores the width and the second channel stores the height. The pupil width is the width of the pupil, and the pupil height is the height of the pupil. The pupil width and the pupil height can be used to determine the size of the pupil. For example, a rectangle is determined as the pupil according to the pupil height and the pupil width, or the pupil height and the pupil width are determined as the short axis and the long axis of the pupil, and an ellipse is determined as the pupil according to the short axis and the long axis.

[0096] According to the coordinates of the pupil center point, the value of the corresponding position in the first channel of the width and height feature map is determined, and this value is determined as the pupil width; according to the coordinates of the pupil center point, the value of the corresponding position in the second channel of the width and height feature map is determined, and this value is determined as the pupil height.

[0097] B4. Determine the pupil position information based on the target pupil center coordinates, pupil width and pupil height.

[0098] Directly use the target pupil center coordinates, pupil width and pupil height as pupil position information; or, determine the coordinates of the vertex or boundary point of the pupil according to the target pupil center coordinates, pupil width and pupil height as pupil position information.

[0099] The image generation method provided by the embodiment of the present invention solves the problem of slow speed and low accuracy when generating images according to event stream data. The event stream within the preset time interval is collected to avoid the problem of high data volume. The preset time interval is divided according to a preset number, the preset number is an integer greater than or equal to 2, and the preset number is determined according to the size of the preset time interval, so as to realize the reasonable division of the time interval, and the motion blur generated when the object moves too fast can be reduced while improving the detection efficiency of the image; by counting the number of all target events generated by the target coordinates of the target event generated in each time period, the pixel coefficient is calculated, and the pixel value of the target coordinate is assigned based on the pixel coefficient, the initial image corresponding to each time period is formed, and each initial image is spliced ​​to form a target image with a preset number of spatial dimensions, thereby improving the accuracy of subsequent image recognition; the image generation method provided by the embodiment of the present application has a relatively simple conversion process, which can effectively retain all effective information, avoid information loss, and improve the conversion speed and accuracy of event stream data to image. The image is recognized and pupil positioning is performed through a deep learning model, and the positioning result is more accurate.

[0100] Embodiment 3

[0101] Figure 5 This is a schematic diagram of the structure of an image generating device provided by Embodiment 3 of the present invention. Figure 5 As shown, the device includes: an event stream acquisition module 41, a time period division module 42, an initial image generation module 43 and a target image determination module 44.

[0102] The event stream acquisition module 41 is used to acquire an event stream within a preset time interval, wherein the events in the event stream include the coordinates and timestamps of event generation;

[0103] A time period division module 42, configured to divide the preset time interval into at least two time periods;

[0104] The initial image generation module 43 is used to count all target events in each time period, determine the target coordinates that generate the target events and the number of target events generated by each target coordinate, determine the pixel value of each target coordinate according to the number of target events generated by each target coordinate, and form an initial image corresponding to the time period;

[0105] The target image determination module 44 is used to determine the target image according to the initial images corresponding to each of the time periods.

[0106] The image generation device provided by the embodiment of the present invention solves the problems of slow speed and low accuracy when generating images based on event stream data. By collecting the event stream within a preset time interval, the problem of high data volume can be avoided, and the preset time interval is divided to obtain at least two time periods. For each time period, the target coordinates of the target event generated within the time period are counted, and the number of all target events generated by the target coordinates is counted. The pixel values ​​of the target coordinates are assigned according to the number of target events to form an initial image corresponding to this time period. The target image is formed based on the initial images corresponding to each time period, and the conversion of the event stream to the image is completed. The conversion process is relatively simple, and all valid information can be effectively retained to avoid information loss, thereby improving the conversion speed and accuracy of the event stream data to the image. Dividing the data into at least two sections for processing can reduce the motion blur caused by the object moving too fast, and further improve the image detection efficiency.

[0107] Optionally, the time period division module 42 is specifically configured to: divide the preset time interval according to a preset number to obtain a time period, where the preset number is an integer and is greater than or equal to 2;

[0108] The preset number is determined according to the size of the preset time interval.

[0109] Optionally, the initial image generation module 43 includes:

[0110] A quantity counting unit, used for counting the number of target events generated by each of the target coordinates, and determining a maximum number and a minimum number;

[0111] a pixel coefficient determination unit, configured to calculate a first difference between the maximum number and the minimum number, and determine a ratio of a preset gray value to the first difference as a pixel coefficient;

[0112] a pixel value determination unit, configured to calculate, for each target coordinate, a second difference between the number of target events generated by the target coordinate and the minimum number, and determine the product of the second difference and the pixel coefficient as the pixel value of the target coordinate;

[0113] The initial image forming unit is used to form an initial image corresponding to the time period based on the pixel values ​​corresponding to each of the target coordinates.

[0114] Optionally, the target image determination module 44 is specifically used to: sequentially splice the initial images according to coordinates to obtain a target image with a preset number of spatial dimensions.

[0115] Optionally, the event stream includes eye features; the device further includes:

[0116] A model input module, used to input the target image into a pre-trained deep learning model to obtain a center point feature map, an offset feature map, and a width and height feature map output by the deep learning model;

[0117] The pupil positioning module is used to determine the position information of the pupil based on the center point feature map, the offset feature map and the width and height feature map.

[0118] Optional, pupil localization module, including:

[0119] A pupil center point determination unit, used to perform a maximum pooling operation on the center point feature map, and determine the coordinates of the maximum pixel value in the center point feature map after pooling as the coordinates of the pupil center point;

[0120] a target coordinate determination unit, configured to correct the coordinates of the pupil center point according to the offset feature map to obtain the target pupil center point coordinates;

[0121] a pupil width and height determination unit, configured to determine the value of the position corresponding to the coordinates of the pupil center point in the first channel of the width and height feature map as the pupil width, and determine the value of the position corresponding to the coordinates of the pupil center point in the second channel of the width and height feature map as the pupil height;

[0122] The pupil positioning unit is used to determine the position information of the pupil based on the target pupil center point coordinates, pupil width and pupil height.

[0123] Optionally, the target coordinate determination unit is specifically used to: determine the value of the position corresponding to the coordinates of the pupil center point in the first channel of the offset feature map as the horizontal coordinate offset, and determine the sum of the horizontal coordinate offset and the horizontal coordinate in the coordinates of the pupil center point as the corrected horizontal coordinate; determine the value of the position corresponding to the coordinates of the pupil center point in the second channel of the offset feature map as the vertical coordinate offset, and determine the sum of the vertical coordinate offset and the vertical coordinate in the coordinates of the pupil center point as the corrected vertical coordinate; determine the target pupil center point coordinates according to the corrected horizontal coordinate and the corrected vertical coordinate.

[0124] The image generating device provided in the embodiment of the present invention can execute the image generating method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0125] Embodiment 4

[0126] Figure 6A schematic diagram of an electronic device 50 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0127] like Figure 6 As shown, the electronic device 50 includes at least one processor 51, and a memory connected to the at least one processor 51 in communication, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 51 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 52 or the computer program loaded from the storage unit 58 to the random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0128] A number of components in the electronic device 50 are connected to the I / O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a disk, an optical disk, etc.; and a communication unit 59, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0129] The processor 51 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 51 executes the various image generation methods and processes described in Embodiment 1 and / or Embodiment 2 above.

[0130] In some embodiments, the image generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 50 via the ROM 52 and / or the communication unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the image generation method described in Embodiment 1 and / or Embodiment 2 above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to perform the image generation method in any other appropriate manner (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0136] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image generation method, characterized in that: include: Acquire an event stream within a preset time interval, wherein the events in the event stream include coordinates and timestamps of event generation; Dividing the preset time interval into at least two time periods; For each time period, all target events within the time period are counted, the target coordinates generating the target events and the number of target events generated by each target coordinate are determined, the pixel value of each target coordinate is determined according to the number of target events generated by each target coordinate, and an initial image corresponding to the time period is formed; The target image is determined according to the initial images corresponding to the time periods.

2. The method according to claim 1, characterized in that The step of dividing the preset time interval into at least two time periods includes: Divide the preset time interval according to a preset number to obtain a time period, where the preset number is an integer and is greater than or equal to 2; The preset number is determined according to the size of the preset time interval.

3. The method according to claim 1, characterized in that Determining the pixel value of each target coordinate according to the number of target events generated by each target coordinate and forming an initial image corresponding to the time period includes: Counting the number of target events generated by each of the target coordinates, and determining a maximum number and a minimum number; Calculating a first difference between the maximum number and the minimum number, and determining a ratio of a preset grayscale value to the first difference as a pixel coefficient; For each target coordinate, calculating a second difference between the number of target events generated by the target coordinate and the minimum number, and determining the product of the second difference and the pixel coefficient as the pixel value of the target coordinate; Based on the pixel values ​​corresponding to the target coordinates, an initial image corresponding to the time period is formed.

4. The method according to claim 2, characterized in that: The determining of the target image according to the initial images corresponding to the time periods includes: The initial images are sequentially spliced ​​according to coordinates to obtain a target image with a preset number of spatial dimensions.

5. The method according to any one of claims 1 to 4, characterized in that: The event stream includes eye features; the method further includes: Inputting the target image into a pre-trained deep learning model to obtain a center point feature map, an offset feature map, and a width and height feature map output by the deep learning model; The position information of the pupil is determined based on the center point feature map, the offset feature map and the width and height feature map.

6. The method according to claim 5, characterized in that The determining the pupil position information based on the center point feature map, the offset feature map and the width and height feature map includes: Performing a maximum pooling operation on the center point feature map, and determining the coordinates of the maximum pixel value in the center point feature map after pooling as the coordinates of the pupil center point; Correcting the coordinates of the pupil center point according to the offset feature map to obtain the coordinates of the target pupil center point; Determine the value of the position corresponding to the coordinates of the pupil center point in the first channel of the width-height feature map as the pupil width, and determine the value of the position corresponding to the coordinates of the pupil center point in the second channel of the width-height feature map as the pupil height; The pupil position information is determined based on the target pupil center point coordinates, pupil width and pupil height.

7. The method according to claim 6, characterized in that The step of correcting the coordinates of the pupil center point according to the offset feature map to obtain the coordinates of the target pupil center point includes: Determine the value of the position corresponding to the coordinate of the pupil center point in the first channel of the offset feature map as the horizontal coordinate offset, and determine the sum of the horizontal coordinate offset and the horizontal coordinate in the coordinate of the pupil center point as the corrected horizontal coordinate; Determine the value of the position corresponding to the coordinate of the pupil center point in the second channel of the offset feature map as the ordinate offset, and determine the sum of the ordinate offset and the ordinate in the coordinate of the pupil center point as the corrected ordinate; The target pupil center point coordinates are determined according to the corrected abscissa and the corrected ordinate.

8. An image generating device, characterized in that: include: An event stream acquisition module, used to acquire an event stream within a preset time interval, wherein an event in the event stream includes a coordinate and a timestamp of the event; A time period division module, used for dividing the preset time interval into at least two time periods; An initial image generation module is used to count all target events in each time period, determine the target coordinates that generate the target events and the number of target events generated by each target coordinate, determine the pixel value of each target coordinate according to the number of target events generated by each target coordinate, and form an initial image corresponding to the time period; The target image determination module is used to determine the target image according to the initial image corresponding to each of the time periods.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the image generating method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image generation method according to any one of claims 1 to 7 when executed.

Images