Object frequency determination method and device, equipment and storage medium

Abstract

The invention discloses an object frequency determination method, apparatus and device, and a storage medium. The method comprises the steps of obtaining a current frame; aiming at each pixel point in the current frame, the pixel point of the current frame and a reference pixel point of a reference frame are analyzed, whether the pixel point generates an event or not is determined, and the position of the reference pixel point in the reference frame to which the reference pixel point belongs is the same as the position of the pixel point in the current frame; under the condition that the pixel point generates an event, determining the event frequency of the event based on the time used for generating the event; based on the event frequency corresponding to each pixel point in the current frame, clustering the pixel points to obtain an object included in the current frame; and for each object obtained by clustering, determining the object frequency of the object based on the event frequency of the pixel point corresponding to the object. According to the technical scheme, the object frequency is determined, and the mode of determining the object frequency is simpler, more convenient and faster.

Description

Technical Field

[0001] The present invention relates to the field of target frequency detection, and in particular to a method, device, equipment and storage medium for determining the frequency of an object. Background Art

[0002] An event camera can be simply understood as a sensor that "only senses moving objects". Each pixel in an event camera has an independent photoelectric sensor module. When the brightness change at the pixel exceeds the set threshold, event data (sometimes also called pulse data) is generated and output.

[0003] In addition, since all pixels work independently, the data output of the event camera is asynchronous and sparse in space. This is also the biggest difference between event cameras and standard cameras. When changes occur in the real scene, the event camera will generate some pixel-level outputs (i.e. events). An event specifically includes (t, x, y, p), where x and y are the pixel coordinates of the event in 2D space, t is the timestamp of the event, and p is the polarity of the event. However, the current method of calculating the frequency of events of objects based on the content recorded by the event camera is still relatively cumbersome. Summary of the invention

[0004] The present invention provides a method, device, equipment and storage medium for determining object frequency to solve the problem in the prior art that it is cumbersome to calculate the event frequency of an object based on event camera characteristics, making the method of determining the object frequency simpler and faster.

[0005] In a first aspect, an embodiment of the present invention provides a method for determining an object frequency, comprising:

[0006] Get the current frame;

[0007] For each pixel in the current frame, analyzing the pixel in the current frame and a reference pixel in a reference frame to determine whether the pixel generates an event, wherein a position of the reference pixel in the reference frame is the same as a position of the pixel in the current frame;

[0008] When an event occurs at the pixel point, determining an event frequency of the event based on the time taken for the event to occur;

[0009] Clustering the pixels based on the event frequency corresponding to each pixel in the current frame to obtain the objects included in the current frame;

[0010] For each object obtained by clustering, the object frequency of the object is determined based on the event frequency of the pixel points corresponding to the object.

[0011] Optionally, obtaining the current frame includes:

[0012] Acquire an initial data stream, wherein the initial data stream includes multiple frames of grayscale images;

[0013] After filtering the gray part of each grayscale image frame in the initial data stream, the target data stream is obtained;

[0014] A current frame is obtained from the target data stream.

[0015] Optionally, when the current frame is the first frame, analyzing the pixel point of the current frame and the reference pixel point of the reference frame to determine whether the pixel point generates an event includes:

[0016] Taking the nth frame after the current frame in the target data stream as the first reference frame;

[0017] Compare the pixel value of the reference pixel in the first reference frame with the pixel value of the pixel in the current frame to determine whether an event is about to occur;

[0018] If an event is to be generated, the mth frame after the first reference frame is used as a second reference frame, and the pixel value of the reference pixel point in the second reference frame is compared with the pixel value of the pixel point in the current frame to determine whether an event is generated;

[0019] If an event is not about to occur, taking the kth frame after the first reference frame as a new first reference frame, and continuing to determine whether an event is about to occur until an event is about to occur;

[0020] Wherein, m, n and k are positive integers.

[0021] Optionally, comparing the pixel value of the reference pixel in the first reference frame with the pixel value of the pixel in the current frame to determine whether an event is about to occur includes:

[0022] If the pixel value of the reference pixel in the first reference frame is different from the pixel value of the pixel in the current frame, it is determined that an event will occur; otherwise, it is determined that no event will occur.

[0023] Optionally, comparing the pixel value of the reference pixel in the second reference frame with the pixel value of the pixel in the current frame to determine whether an event is generated includes:

[0024] If the pixel value of the reference pixel in the second reference frame is the same as the pixel value of the pixel in the current frame, it is determined that an event has occurred; otherwise, no event has occurred.

[0025] Optionally, when the current frame is not the first frame, analyzing the pixel point and the reference pixel point of the reference frame to determine whether the pixel point generates an event includes:

[0026] Select a reference frame from the reference events of the historical data;

[0027] Based on a comparison result between a pixel value of a reference pixel of the reference frame and a pixel value of the pixel of the current frame, it is determined whether the pixel generates an event.

[0028] Optionally, when the reference frame is the frame with the largest frame interval with the current frame in the reference event, the step of determining whether the pixel generates an event based on a comparison result of a pixel value of a reference pixel of the reference frame and a pixel value of the pixel of the current frame includes:

[0029] If the pixel value of the reference pixel of the reference frame is the same as the pixel value of the pixel of the current frame, it is determined that the pixel generates an event; otherwise, continue to select the next frame after the current frame from the target data stream as the new current frame, and continue to perform the operation of determining whether the pixel generates an event based on the comparison result of the pixel value of the reference pixel of the reference frame with the pixel value of the pixel of the current frame until an event is generated.

[0030] Optionally, determining the event frequency of the event based on the time taken for the event to occur includes:

[0031] Determine the number of frames used to determine whether an event occurs;

[0032] Determine the time taken for the event camera to output the number of frames used;

[0033] The inverse of the elapsed time is determined as the event frequency of the event.

[0034] Optionally, the frequency range formed by the event frequencies of the pixels included in the same object is within a set range, and the object frequency corresponding to each object is the average of the event frequencies of the pixels corresponding to the object.

[0035] Optionally, the method further includes:

[0036] When the used time is less than the set interval threshold, the event frequency of the event is smoothed according to the last used time of the last event and the used time of the event.

[0037] Optionally, the smoothing of the event frequency of the event according to the last time taken for the last event and the time taken for the event includes:

[0038] Determining a weight coefficient based on a ratio of the time taken for the event to the last time taken for the last event;

[0039] Based on the weight coefficient, the event frequencies of the events are weighted and summed to obtain updated event frequencies.

[0040] Optionally, when an event is generated at the pixel point, determining the event frequency of the event based on the time taken for the event to be generated includes:

[0041] In the case where an event is generated at the pixel point, if the number of times the pixel point generates events is greater than the event filtering threshold, the event frequency of the event is determined based on the time taken for the event to be generated; otherwise, continue to determine whether the pixel point generates the next event until the number of times the pixel point generates events is greater than the event filtering threshold.

[0042] In a second aspect, an embodiment of the present invention provides an object frequency determination device, comprising:

[0043] Acquisition module, used to obtain the current frame;

[0044] An analysis module, configured to analyze, for each pixel in the current frame, the pixel in the current frame and the reference pixel in the reference frame to determine whether the pixel generates an event, and the position of the reference pixel in the reference frame is the same as the position of the pixel in the current frame;

[0045] A first determination module is used to determine the event frequency of the event based on the time taken for the event to occur when the event occurs at the pixel point;

[0046] A clustering module, used for clustering the pixels in the current frame based on the event frequency corresponding to each pixel in the current frame to obtain the objects included in the current frame;

[0047] The second determination module is used to determine the object frequency of each object obtained by clustering based on the event frequency of the pixel points corresponding to the object.

[0048] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0049] at least one processor; and

[0050] a memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the object frequency determination method described in any embodiment of the present invention.

[0052] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the object frequency determination method described in any embodiment of the present invention when executed.

[0053] The technical solution of the embodiment of the present invention is as follows: acquiring a current frame; for each pixel in the current frame, analyzing the pixel in the current frame and the reference pixel in the reference frame to determine whether the pixel generates an event, wherein the position of the reference pixel in the reference frame is the same as the position of the pixel in the current frame; in the case where the pixel generates an event, determining the event frequency of the event based on the time taken for the event to be generated; clustering the pixels based on the event frequencies corresponding to the pixels in the current frame to obtain the objects included in the current frame; for each clustered object, determining the object frequency based on the event frequency of the pixel corresponding to the object; solving the problem in the prior art that it is cumbersome to calculate the event frequency of an object based on the characteristics of an event camera, determining whether an event is generated by analyzing the pixel in the current frame and the pixel in the reference frame, and determining the object frequency by clustering the event frequencies, so that the method of determining the object frequency is simpler and faster.

[0054] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 A flow chart of a method for determining an object frequency provided in Embodiment 1 of the present invention;

[0057] Figure 2A A flowchart for determining whether a pixel point generates an event when the current frame is the first frame provided in the second embodiment of the present invention;

[0058] Figure 2BA flowchart for determining whether a pixel point generates an event when the current frame is not the first frame, provided in Embodiment 3 of the present invention;

[0059] Figure 3 A flowchart of a method for determining an object frequency provided in Embodiment 4 of the present invention;

[0060] Figure 4 A flowchart of a method for determining an object frequency provided in Embodiment 4 of the present invention;

[0061] Figure 5 A schematic diagram of the structure of an object frequency determination device provided in Embodiment 5 of the present invention;

[0062] Figure 6 A schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0065] Embodiment 1

[0066] Figure 1This is a flow chart of a method for determining an object frequency provided in the first embodiment of the present invention. This embodiment is applicable to the case of determining an object frequency. The method can be performed by an object frequency determination device. The object frequency determination device can be implemented in the form of hardware and / or software. The object frequency determination device can be configured in an electronic device. In an application scenario, the object frequency of an object in a shooting area of ​​an event camera, such as a dynamic vision sensor (DVS) camera, can be calculated and matched with the object itself. For example, the matching of an LED light (i.e., the object itself) and a light spot (i.e., the object in the shooting area of ​​the event camera). Specifically, when a user wears a head-mounted display device, the LED light forms a light spot in the user's eyes. The light spots corresponding to LED lights of different frequencies have different object frequencies. By determining the object frequency, it is matched with the LED lights in the actual environment to determine the correspondence between the LED light and the light spot. Figure 1 As shown, the method includes:

[0067] S110, obtaining the current frame.

[0068] Among them, the current frame is a frame image currently being analyzed, and the image can be any image, such as a black and white image or a grayscale image. In this embodiment, multiple frames of images included in the data stream can be used as the current frame in sequence. The current frame can be the first frame or a non-first frame.

[0069] In this embodiment, the image may include an object. In this step, the current frame is acquired to analyze the current frame and determine the frequency of the object in the current frame.

[0070] In this embodiment, the current frame can be obtained from the data stream output by the event camera, and each frame image in the data stream can be used as the current frame in turn.

[0071] S120. For each pixel in the current frame, analyze the pixel in the current frame and the reference pixel in the reference frame to determine whether the pixel generates an event, and the position of the reference pixel in the reference frame is the same as the position of the pixel in the current frame.

[0072] In this embodiment, a pixel may be the smallest unit in a digital image (such as the smallest image unit), representing an image unit with a fixed position and a specific color or brightness value. A pixel may be considered as a very small point that constitutes an image, and an image may be composed of many pixels.

[0073] The reference frame can be frame data used as a reference when analyzing the current frame, and the reference frames corresponding to different current frames may not be exactly the same. Among them, if the current frame is the first frame, the reference frame can only be selected from the data stream after the first frame; if the current frame is not the first frame, it can be selected from the historical data before the current frame. The reference frame can be selected according to the actual situation, and the present invention does not impose any restrictions on this. The historical data can be the data recorded for the last time or the last x times an event occurred, where x is a positive integer. Historical events include data frames corresponding to production events.

[0074] The reference pixel point may be a pixel point in the reference frame that has the same position as a pixel point in the current frame.

[0075] The position of the reference pixel in the reference frame is the same as the position of the pixel in the current frame. It can be considered that the reference pixel and the pixel in the current frame are the same pixel in different frames. Events can be determined by the change pattern of the pixel value. The event camera can generate an event by detecting the brightness change of each pixel. For example, an event can be understood as the process in which the pixel value of the same pixel in the image changes from the same to different and then to the same. For example, an event can be understood as the process in which the pixel value of the same pixel in the image changes from different to the same. For example, from 0 to 255 and then to 0 is an event.

[0076] After obtaining the current frame, this step can analyze each pixel in the current frame in turn to determine whether the current pixel generates an event. The analysis method is not limited here, such as analyzing the change of pixel values ​​in multiple frames to determine whether an event is generated.

[0077] In this embodiment, each pixel in the current frame and the reference pixel in the reference frame with the same position as the pixel in the current frame are analyzed to determine whether the pixel value of the pixel in the current frame has undergone a process of changing from the same to different and then to the same. If the process has occurred, it is determined that an event has occurred. For example, if the pixel value of the pixel changes to (0, 255, 0) or (255, 0, 255), etc., it can be determined that the pixel has generated an event. If the process has not occurred, it is determined that no event has occurred. For example, if the pixel value of the pixel changes to (0, 255, 255) or (255, 0, 0) or (0, 0, 255) or (255, 255, 0), etc., it can be determined that the pixel has not generated an event.

[0078] S130: When an event occurs at the pixel point, determine the event frequency of the event based on the time taken for the event to occur.

[0079] Among them, the output of the event camera includes the timestamp of each frame, so the time taken for the event to occur can be the sum of the time for the number of frames corresponding to the event generated by the pixel point; specifically, in general, the event camera outputs at a certain frequency, and the time it takes for the event camera to generate one frame is fixed, so the number of frames required to generate an event multiplied by the time taken for one frame can be used to obtain the time taken for the event to occur. Event frequency can refer to the frequency of events. In the case of events generated by pixels, the time taken for the event to occur can be determined based on the frequency of the event camera, and the frequency of events can be determined based on the time taken for the events to occur.

[0080] S140: Clustering the pixels based on event frequencies corresponding to the pixels in the current frame to obtain objects included in the current frame.

[0081] Clustering may refer to the process of dividing a collection of physical or abstract objects into multiple classes consisting of similar objects, and the present invention does not limit the clustering algorithm. Similarly, there is no limit on the objects included in the current frame obtained, which can be determined based on the actual image.

[0082] In this embodiment, the pixels are clustered based on the event frequency corresponding to each pixel in the current frame. The pixels whose event frequency in the current frame is within a set range are clustered, and the area of ​​pixels whose event frequency in the current frame is within the set range is considered to be the same object.

[0083] S150 . For each object obtained through clustering, determine the object frequency of the object based on the event frequency of the pixel points corresponding to the object.

[0084] The event frequency of a pixel point may be understood as the frequency of events generated by the pixel point, and the object frequency may be the frequency of objects in the current frame.

[0085] This embodiment does not limit how to determine the object frequency based on the event frequency of the pixels included in the object. For example, the object frequency can be obtained by performing mathematical operations on the event frequency of the pixels included in the object. For example, the object frequency can be determined based on the event frequency of the pixels included in each object. The means for determining the object frequency by the event frequency of the pixels included in the object is not limited. For example, the mean of the event frequency of the pixels included in the object can be used as the object frequency, or the event frequency with the largest value among the event frequencies of the pixels included in the object can be selected as the object frequency, or the median of the event frequencies of the pixels included in the object can be selected as the object frequency.

[0086] In one embodiment, for each object obtained by clustering, the object frequency of the object may be obtained by calculating the average of the event frequencies of the pixels in the area corresponding to the object.

[0087] In one embodiment, after determining the object frequency of each object obtained by clustering, the object may also be characterized by the center point of the area determined by each pixel point in the object.

[0088] In this embodiment, the pixels of the current frame and the pixels of the reference frame can be analyzed to determine whether an event has occurred, and the object frequency can be determined by event frequency clustering. This solves the problem of cumbersome calculation of the event frequency of objects based on event camera characteristics in the prior art, and makes the method of determining the object frequency simpler and faster.

[0089] Optionally, obtaining the current frame includes:

[0090] Acquire an initial data stream, wherein the initial data stream includes multiple frames of grayscale images;

[0091] After filtering the gray part of each grayscale image frame in the initial data stream, the target data stream is obtained;

[0092] A current frame is obtained from the target data stream.

[0093] Specifically, the event camera can output a frame data stream, wherein the initial data stream can be a frame data stream directly output by the event camera, wherein the initial data stream includes multiple frames of grayscale images. In order to further simplify the calculation amount of the event frequency, it is necessary to filter the gray part of each frame of the grayscale image in the initial data stream, and the specific filtering method is not limited in the present invention; after filtering, the target data stream is obtained, that is, the data stream formed by the black and white image; thereby, the current frame is obtained from the target data stream, and the current frame is a black and white image.

[0094] In this embodiment, the target data stream is obtained by filtering the initial data stream, and the current frame is obtained from the initial data stream. The current frame is a black and white image, which further simplifies the calculation amount of the event frequency.

[0095] Embodiment 2

[0096] Figure 2A A flowchart for determining whether a pixel generates an event when the current frame is the first frame is provided for the second embodiment of the present invention; the technical solution of this embodiment is further refined on the basis of the above embodiment. The specific implementation process of analyzing the pixel of the current frame and the reference pixel of the reference frame to determine whether the pixel generates an event is described.

[0097] like Figure 2A As shown, if the current frame is the first frame, the method of this embodiment may include the following steps:

[0098] S210a, obtain the current frame.

[0099] S220a: Use the nth frame after the current frame in the target data stream as the first reference frame.

[0100] Specifically, since the current frame is the first frame, the reference frame to be compared with it can only be selected from the subsequent frames of the target data stream. n is a positive integer, such as n is 1, and this step uses the first frame after the current frame in the target data stream as the first reference frame.

[0101] S230a. Compare the pixel value of the reference pixel in the first reference frame with the pixel value of the pixel in the current frame to determine whether an event is about to occur.

[0102] The event that is about to occur can be understood as the probability of the pixel point generating an event being relatively high or the possibility of the pixel point generating an event being relatively high, but the event has not yet occurred. If, through the analysis of the pixel point, it is determined that the change process of the pixel point in multiple frames is about to generate an event, it can be considered that the probability of the pixel point generating an event is relatively high.

[0103] When determining an event, this embodiment first selects the current frame, and then selects a pixel from the current frame for analysis. The same operation can be performed on each pixel in the current frame. For a pixel in the current frame, the change rule of the pixel value of the pixel and the reference pixel in the first reference frame is determined, and whether an event will occur is determined based on the change rule. If the pixel value of the reference pixel in the first reference frame is different from the pixel value of the pixel in the current frame, it can be considered that an event will occur.

[0104] S240a. If an event is about to occur, the mth frame after the first reference frame is used as the second reference frame, and the pixel value of the reference pixel point in the second reference frame is compared with the pixel value of the pixel point in the current frame to determine whether an event occurs.

[0105] m is a positive integer, where m≥n≥1. If m is 1 and n is 1, this step uses the first frame after the current frame in the target data stream as the first reference frame and the first frame after the first reference frame as the second reference frame.

[0106] Specifically, the pixel value of the pixel point in the current frame is compared again with the pixel value of the reference pixel point in the second reference frame to determine whether an event occurs.

[0107] In this embodiment, after determining that an event is about to occur, for a pixel point in the current frame that is about to generate an event, a change rule between the pixel value of the pixel point and the reference pixel point in the second reference frame is determined, and whether an event is generated is determined based on the change rule. If the pixel value of the reference pixel point in the second reference frame is the same as the pixel value of the pixel point in the current frame, it can be considered that an event has occurred.

[0108] S250a: If an event is not about to occur, use the kth frame after the first reference frame as a new first reference frame, and continue to determine whether an event is about to occur until an event is about to occur.

[0109] If the pixel value of the pixel point in the current frame is compared with the pixel value of the reference pixel point in the first reference frame, and it is determined that no event is about to occur; then the kth frame after the first reference frame is used as the new first reference frame, and the comparison is continued to determine whether an event is about to occur, until it is determined that an event is about to occur.

[0110] Among them, m, n and k are positive integers. For example, k is 1.

[0111] S260a: When an event occurs at the pixel point, determine the event frequency of the event based on the time taken for the event to occur.

[0112] S270a: Cluster the pixels based on the event frequency corresponding to each pixel in the current frame to obtain objects included in the current frame.

[0113] S280a. For each object obtained through clustering, determine the object frequency of the object based on the pixel frequency of the pixel points corresponding to the object.

[0114] In the present embodiment, when the current frame is the first frame, by analyzing the pixel points of the current frame and the reference pixel points of the reference frame, it is first determined whether an event will occur, and then it is determined whether an event occurs, thereby achieving accurate determination of the event corresponding to the current frame being the first frame.

[0115] Optionally, comparing the pixel value of the reference pixel in the first reference frame with the pixel value of the pixel in the current frame to determine whether an event is about to occur includes:

[0116] If the pixel value of the reference pixel in the first reference frame is different from the pixel value of the pixel in the current frame, it is determined that an event will occur; otherwise, it is determined that no event will occur.

[0117] In this embodiment, whether an event is about to occur can be determined based on whether the pixel value of the reference pixel in the first reference frame is different from the pixel value of the pixel in the current frame; for example, if the pixel value of the pixel in the current frame is 0 and the pixel value of the reference pixel in the first reference frame is 255, and the two are different, it can be determined that an event is about to occur; if the pixel value of the pixel in the current frame is 0 and the pixel value of the reference pixel in the first reference frame is also 0, and the two are the same, it can be determined that no event is about to occur.

[0118] Optionally, comparing the pixel value of the reference pixel in the second reference frame with the pixel value of the pixel in the current frame to determine whether an event is generated includes:

[0119] If the pixel value of the reference pixel in the second reference frame is the same as the pixel value of the pixel in the current frame, it is determined that an event has occurred; otherwise, no event has occurred.

[0120] In this embodiment, only when it is determined that an event is about to occur can it be further determined whether an event is to occur based on whether the pixel value of the reference pixel in the second reference frame is the same as the pixel value of the pixel in the current frame; for example, if the pixel value of the pixel in the current frame is 0, and the pixel value of the reference pixel in the first reference frame is 255, and the difference between the two determines that an event is to occur; then the pixel value 0 of the pixel in the current frame is compared with the pixel value 0 of the reference pixel in the second reference frame again, and if the two are the same, the process of generating an event (0, 255, 0) is satisfied, and it is determined that an event has occurred; if the pixel value 0 of the pixel in the current frame is compared with the pixel value 255 of the reference pixel in the second reference frame again, and if the two are different, the process of generating an event (0, 255, 255) is not satisfied, and it is determined that no event has occurred, then a new second reference frame is selected until an event occurs.

[0121] In the present embodiment, when the current frame is the first frame, a determination is made as to whether an event will occur by comparing the pixel value with the reference pixel point in the first reference frame; after determining that an event will occur, a determination is made as to whether an event will occur by comparing the pixel value with the reference pixel point in the second reference frame. Whether an event will occur is determined by comparing only twice whether it is different or the same, thereby simplifying the comparison means and improving the determination efficiency.

[0122] Embodiment 3

[0123] Figure 2B A flowchart for determining whether a pixel generates an event when the current frame is not the first frame is provided for the third embodiment of the present invention; the technical solution of this embodiment is further refined on the basis of the above embodiment. The specific implementation process of analyzing the pixel of the current frame and the reference pixel of the reference frame to determine whether the pixel generates an event is described.

[0124] like Figure 2B As shown, if the current frame is not the first frame, the method of this embodiment may include the following steps:

[0125] S210b, obtain the current frame.

[0126] S220b, selecting a reference frame from the reference events of the historical data;

[0127] The historical data may be considered as data associated with a historical event, such as the historical data including the data frame involved in the historical event and the pixel value of the pixel point where the event occurred in the data frame.

[0128] In one embodiment, since the current frame is not the first frame, the data frame in which the event occurred before the current frame can be determined as historical data; the reference event can be understood as an event that has occurred in the historical data, such as the last event generated when analyzing the current frame.

[0129] In this embodiment, when judging the current frame, the pixel values ​​of the pixels can be selected from the reference events of the historical data to generate different frames as reference frames. Among them, the pixel values ​​of the pixels generating different frames can be considered to be different from the pixel values ​​of the current frame selected when judging whether the event is generated in the process of judging whether the event is generated. For example, when judging whether the current frame generates an event, the pixel values ​​of the pixels generating different frames are the frames with different pixel values ​​from the pixel values ​​of the current frame selected for the first time. For example, if the current frame is the first frame and the pixel values ​​of the pixels of the current frame are 0, the second frame can be compared with the current frame. If the pixel value at the pixel position of the second frame corresponding to the current frame is 255, the second frame is a frame with different pixel values ​​of the pixels. If the pixel value at the pixel position of the second frame corresponding to the current frame is 0, the pixel value of the reference pixel of the second frame, i.e., the first reference frame, is not different from the pixel value of the pixel of the current frame. Then the third frame is selected as the first reference frame. If the pixel value of the reference pixel of the third frame is 255, the third frame is a frame with different pixel values ​​of the pixels. Otherwise, the same applies. The pixel value of the pixel point that produces different frames can be 255, and the process of generating an event is changing from 0 to 255, or the pixel value of the pixel point that produces different frames can be 0, and the process of generating an event is changing from 255 to 0.

[0130] S230b: Determine whether an event occurs at the pixel point based on a comparison result between a pixel value of the reference pixel point of the reference frame and a pixel value of the pixel point of the current frame.

[0131] Specifically, since the pixel values ​​of the pixels in the reference event of the historical data are selected to generate different frames as reference frames. Only two reference frames can be saved in the historical data, and these two reference frames have been compared in the reference event of the historical data. The pixel values ​​of the reference pixels of the two reference frames are different, for example, the pixels can be 0 and 255 respectively. In this step, the pixel values ​​of the reference pixels of a reference frame selected from the reference frame can be compared with the pixel values ​​of the pixels of the current frame, and whether the pixel generates an event can be determined based on the comparison result; whether they are the same can be determined based on the different frame, or whether they are different can be determined based on the same frame; for example, if the process of the last event generation is 0-255-0, the frame corresponding to 255 can be retained in the historical data as a reference frame (which can correspond to the first reference frame when the event is about to be generated), and the frame corresponding to the second 0 (which can correspond to the second reference frame when the event is about to be generated) can be used as a reference frame, and the pixel values ​​of the pixels of the current frame can be compared with 255 or with 0. If compared with 255, the pixel values ​​of the pixels of the current frame must be the same as 255, and if compared with 0, the pixels of the current frame must be different.

[0132] S240b: When an event occurs at the pixel point, determine the event frequency of the event based on the time taken for the event to occur.

[0133] S250b: clustering the pixels based on the event frequency corresponding to each pixel in the current frame to obtain objects included in the current frame.

[0134] S260b: For each object obtained by clustering, determine the object frequency of the object based on the pixel frequency of the pixel points corresponding to the object.

[0135] In the present embodiment, when the current frame is not the first frame, by selecting at least two different reference frames from the reference events of historical data, the pixel values ​​of the reference pixels of the reference frames are compared with the pixel values ​​of the pixels of the current frame, and based on the result of one comparison, it is possible to determine whether an event has been generated at the pixel point, thereby realizing the determination of the corresponding event generated when the current frame is not the first frame.

[0136] Optionally, when the reference frame is the frame with the largest frame interval with the current frame in the reference event, the step of determining whether the pixel generates an event based on a comparison result of a pixel value of a reference pixel of the reference frame and a pixel value of the pixel of the current frame includes:

[0137] If the pixel value of the reference pixel of the reference frame is the same as the pixel value of the pixel of the current frame, it is determined that the pixel generates an event; otherwise, continue to select the next frame after the current frame from the target data stream as the new current frame, and continue to perform the operation of determining whether the pixel generates an event based on the comparison result of the pixel value of the reference pixel of the reference frame with the pixel value of the pixel of the current frame until an event is generated.

[0138] Specifically, at least two reference frames are selected from the reference events of historical data, and they must be in order. In this embodiment, the frame with the largest frame interval with the current frame in the reference event can be selected as the reference frame; if the pixel value of the reference pixel of the reference frame is the same as the pixel value of the pixel of the current frame, it is determined that the pixel has generated an event; for example, if the pixel value of the pixel changes to (0, 255, 0, 255), the first, second, and third frames can generate an event, and the second, third, and fourth frames can also generate an event. When determining whether an event is generated with the first frame as the current frame, the second frame is used as the first reference frame, that is, the first frame (0) as the current frame and the second frame (255) as the first reference frame can be compared. If the pixel values ​​are different, it can be determined that the reference pixel of the second frame will generate an event; if the first frame as the current frame is 0 and the second frame as the first reference frame is also 0, the pixel value change is (0, 0), the current frame (0) and the first reference frame (0) have the same pixel value, it is determined that the pixel will not generate an event, and the first frame after the first reference frame is compared. The third frame is used as a new first reference frame, and is continuously compared with the first frame as the current frame to determine whether an event is about to occur, until an event is about to occur; after determining that the pixel point is about to generate an event, the frame after the first reference frame where the pixel point is determined to generate an event can be used as the second reference frame, and the current frame is compared with the second reference frame. If the pixel value of the pixel point changes to (0, 255, 0, 255), the current frame is the first frame (0), the first reference frame is the second frame (255), it is determined that an event is about to occur, and the third frame (0) is used as the second reference frame. Compared with the current frame (0), if the pixel values ​​are the same, it can be determined that the pixel point has generated an event. If the pixel value of the third frame is 255, the pixel value change of the pixel point is (0, 255, 255). The current frame is different from the second reference frame, then it can be determined that the pixel point has not generated an event. Then continue to select subsequent frames as the second reference frame until an event occurs. Store historical data. If the pixel value change of the pixel point is (0, 255, 0, 255), the first, second, and third frames can generate an event. The second and third frames can be stored in the historical data as reference frames, and the fourth When a frame is used as the current frame to determine whether an event occurs, it is compared with the second frame or the third frame. The second frame generates different frames for the pixel values ​​of the pixels. If the comparison result of the fourth frame and the second frame is the same, then the second, third, and fourth frames can also generate an event (i.e., the second frame with the largest frame interval of the current frame is used as the reference frame (255) and compared with the fourth frame (255) of the current frame. If the pixel values ​​are the same, it can be determined that the pixel generates an event). Therefore, the third frame and the fourth frame can be stored as reference frames in the historical data, and the second frame can be deleted from the historical data or retained. Different events can correspond to different reference frames, which are used to determine whether subsequent events occur.

[0139] If the current frame, i.e., the fourth frame, is 0, the pixel value of the pixel point changes to (0, 255, 0, 0), and the second frame (255) is different from the fourth frame (0), it can be determined that the pixel point has not generated an event, and then continue to select the next frame after the current frame from the target data stream, i.e., the fifth frame as the new current frame, and continue to perform the comparison result based on the pixel value of the reference pixel point of the reference frame and the pixel value of the pixel point of the fifth frame as the current frame to determine whether an event has occurred. If the fifth frame is 255, the pixel value of the pixel point changes to (0, 255, 0, 0, 255), and the pixel value of the reference pixel point of the second frame is the same as the pixel value of the pixel point of the fifth frame, then it is considered that an event has occurred. The third frame and the fifth frame can be further used as reference frames, and the sixth frame, the next frame after the current frame, is selected as the new current frame. If the sixth frame is 0, by comparing the third frame (0) with the sixth frame (0), the two are the same, and the pixel value of the pixel point changes to (0, 255, 0, 0, 255, 0), it can be determined that the pixel point has generated an event. If the sixth frame is still 255, continue to select the next frame after the current frame from the target data stream as the new current frame, and continue to perform the operation of determining whether the pixel point generates an event based on the comparison result of the pixel value of the reference pixel point of the reference frame and the pixel value of the pixel point, until an event is generated.

[0140] In the present embodiment, when the current frame is not the first frame, by flexibly selecting a reference frame, the pixel value of the reference pixel of the reference frame is compared with the pixel value of the pixel of the current frame, and whether an event is generated at the pixel point can be determined based on the result of one comparison, thereby simplifying the comparison operation and realizing the determination of the corresponding event generated when the current frame is not the first frame.

[0141] A method for determining an object frequency provided by a second embodiment of the present invention distinguishes whether the current frame is the first frame and executes different event determination methods according to different current frames. This not only can flexibly determine whether an event occurs, but also simplifies the determination means.

[0142] Embodiment 4

[0143] Figure 3 This is a flowchart of a method for determining the frequency of an object provided in the fourth embodiment of the present invention. The technical solution of this embodiment is further expanded and refined on the basis of the above embodiments. It not only further describes the process of determining the event frequency of an event based on the time taken for the event to occur when an event occurs at a pixel point, but also includes: after determining the event frequency of an event, when the time taken is less than a set interval threshold, the event frequency of the event is smoothed according to the last time taken for the last event and the time taken for the event.

[0144] like Figure 3 As shown, the method of this embodiment may include the following steps:

[0145] S310, obtaining the current frame.

[0146] S320. For each pixel in the current frame, analyze the pixel in the current frame and the reference pixel in the reference frame to determine whether the pixel generates an event, and the position of the reference pixel in the reference frame is the same as the position of the pixel in the current frame.

[0147] S330: When an event occurs at the pixel point, determine the event frequency of the event based on the time taken for the event to occur.

[0148] Optionally, when an event is generated at the pixel point, determining the event frequency of the event based on the time taken for the event to be generated includes:

[0149] In the case where an event is generated at the pixel point, if the number of times the pixel point generates events is greater than the event filtering threshold, the event frequency of the event is determined based on the time taken for the event to be generated; otherwise, continue to determine whether the pixel point generates the next event until the number of times the pixel point generates events is greater than the event filtering threshold.

[0150] Among them, the event filtering threshold can be a pre-set threshold for filtering intermediate events, which is not limited in the present invention. Specifically, in the case where a pixel generates an event, if the number of times the pixel generates an event is greater than the event filtering threshold, the event frequency of the event is determined based on the time taken for the event to be generated; for example, if the pixel value of the event generated by the pixel changes to (255, 0, 255, 0, 255), there are five frames in total, and there are three events in these five frames, namely the first, second, and third frames (255, 0, 255), the second, third, and fourth frames (0, 255, 0), and the third, fourth, and fifth frames (255, 0, 255), wherein the number of times the pixel generates an event is 3 times, and the event filtering threshold is 2, then the intermediate events of the second, third, and fourth frames (0, 255, 0) are filtered out to determine the final event frequency. If the number of times a pixel generates events is less than or equal to the event filtering threshold, it is necessary to continue to determine whether the pixel generates the next event until the number of times the pixel generates events is greater than the event filtering threshold, and then filter.

[0151] In this embodiment, events generated by pixel points are filtered using an event filtering threshold, so that the calculation result of the event frequency is smoother.

[0152] Optionally, determining the event frequency of the event based on the time taken for the event to occur includes:

[0153] Determine the number of frames used to determine whether an event occurs;

[0154] Determine the time taken for the event camera to output the number of frames used;

[0155] The inverse of the elapsed time is determined as the event frequency of the event.

[0156] Specifically, the number of frames used to generate an event can be understood as the number of frames used when analyzing whether an event is generated, where at least three frames are required to generate an event; generally, the event camera outputs at a certain frequency, and the time it takes for the event camera to generate one frame is fixed, so the time used to determine the number of frames used by the event camera output can be obtained by multiplying the number of frames by the time used for one frame. The inverse of the time used is determined as the event frequency of the event, which can be calculated by: hz = 1 / time used for the number of frames used; where hz is the event frequency.

[0157] In this embodiment, by determining the inverse of the time taken to generate an event as the event frequency of the event, effective judgment and calculation of the frequency is achieved.

[0158] S340: When the used time is less than the set interval threshold, smooth the event frequency of the event according to the previous used time of the previous event and the used time of the event.

[0159] The interval threshold can be set according to actual conditions. The event time can refer to the time used to generate the current event, and the previous time used for the previous event can refer to the time of the previous event generated before the current event. Smoothing processing can generally be used to remove noise, reduce fluctuations, eliminate outliers, etc., to optimize data.

[0160] Specifically, when the time used is less than the set interval threshold, it can be understood that if the number of frame intervals used for a pixel point to generate an event is relatively small, less than the set interval threshold, then the event frequency of the event is smoothed according to the previous time used for the previous event and the time used for the event. For example, if the number of frames used to generate the event is three frames, and the time required for three frames is less than a set time interval threshold, then the event frequency of the event is smoothed according to the previous time used for the previous event and the time used for the event; if the time required for the number of frames to generate the event is not less than the set time interval threshold, the event frequency is directly calculated.

[0161] In this embodiment, by determining that the time taken by a pixel point to generate an event is less than a set interval threshold, the event frequency of the event is smoothed according to the previous time taken by the previous event and the time taken by the event, so that the calculation of the event frequency is more accurate.

[0162] Optionally, the smoothing of the event frequency of the event according to the last time taken for the last event and the time taken for the event includes:

[0163] Determining a weight coefficient based on a ratio of the time taken for the event to the last time taken for the last event;

[0164] Based on the weight coefficient, the event frequencies of the events are weighted and summed to obtain updated event frequencies.

[0165] Specifically, the weight coefficient can be set to a. Based on the ratio of the time taken for the event to the previous time taken for the previous event, the weight coefficient can be determined as: f(a)=0.95*t / t1, where t represents the time taken for the current event and t1 represents the time taken for the previous event.

[0166] Based on the weight coefficient, the event frequencies of the events are weighted and summed, and the updated event frequency can be determined by the calculation formula f(hz(t))=a*hz(t-1)+(1-a)hz(t).

[0167] In this embodiment, the event frequency of the event is smoothed by the above formula, so that the calculation of the event frequency can be more accurate.

[0168] S350: Clustering the pixels based on event frequencies corresponding to the pixels in the current frame to obtain objects included in the current frame.

[0169] S360: For each object obtained through clustering, determine the object frequency of the object based on the event frequency of the pixel points corresponding to the object.

[0170] Optionally, the frequency range formed by the event frequencies of the pixels included in the same object is within a set range, and the object frequency corresponding to each object is the average of the event frequencies of the pixels corresponding to the object.

[0171] The frequency range may refer to a limited range of event frequencies, and the pixel points corresponding to all event frequencies within the set range may be clustered into the same object.

[0172] Specifically, the same object can be obtained by clustering the pixel points corresponding to the event frequencies within a set range in the current frame; since the event frequencies of the pixel points may be different; therefore, the object frequency corresponding to each object can be the average value of the event frequencies of the pixel points corresponding to the object.

[0173] In this embodiment, the same object is obtained by clustering pixels whose event frequencies are within a set range, and the object frequency of the object is determined according to the average value of the event frequencies of the pixels in the object, thereby achieving accurate clustering of the objects.

[0174] A method for determining object frequency is provided in a fourth embodiment of the present invention. Events generated by pixel points are filtered through an event filtering threshold, so that the calculation result of the event frequency is smoother; the inverse of the time used to generate the event is determined as the event frequency of the event, thereby achieving effective judgment and calculation of the frequency; when it is determined that the time used by the pixel point to generate the event is less than a set interval threshold, the event frequency of the event is smoothed, so that the calculation of the event frequency is more accurate; the same object is obtained by clustering the pixel points whose event frequencies are within a set range, and the object frequency of the object is determined according to the average value of the event frequencies of the pixel points in the object, thereby achieving accurate clustering of the object.

[0175] Figure 4 This is a flow chart of a method for determining the frequency of an object provided in Embodiment 3 of the present invention. Figure 4 As shown, the process of analyzing the pixels of the current frame and the pixels of the reference frame to determine whether an event occurs and determining the object frequency by clustering the event frequency can be:

[0176] 1. Obtain the frame data stream of the sensor, that is, the data stream formed by the output image (that is, obtain the initial data stream);

[0177] 2. Convert the data stream to use only the areas where the pixel value is 0 or 255 (i.e., filter the gray part of each grayscale image in the initial data stream to obtain the target data stream);

[0178] 3. Determine whether an event has occurred by comparing the difference with the historical data (i.e., select a reference frame from the reference event of the historical data; and determine whether an event has occurred at the pixel point based on the comparison result of the pixel value of the reference pixel point of the reference frame with the pixel value of the pixel point of the current frame);

[0179] 4. If an event occurs, the event frequency is calculated according to the time (i.e., when an event occurs at the pixel point, the event frequency of the event is determined based on the time taken for the event to occur);

[0180] 5. The frequency is calculated as hz = 1 / event generation interval, where the event generation interval can be represented by the time taken for the event to occur;

[0181] 6. In order to calculate more accurately, the values ​​of the same point interval threshold that are too small will be smoothed, using the calculation formula f(hz(t))=a*hz(t-1)+(1-a)hz(t), where the coefficient a is a formula related to the event proportion time, f(a)=0.95*t / t1, where t represents the time taken by the current event, and t1 represents the time taken by the previous event (i.e., when the time taken is less than the set interval threshold, the event frequency of the event is smoothed according to the previous time taken by the previous event and the time taken by the event);

[0182] 7. At the same time, a threshold can be selected to filter events during calculation. For example, after an event at a point occurs for the nth time, the frequency of the first and n+1th events will be calculated, and the nth time will be ignored (that is, when an event occurs at the pixel point, if the number of times the pixel point generates an event is greater than the event filtering threshold, the event frequency of the event is determined based on the time taken for the event to occur; otherwise, continue to determine whether the pixel point generates the next event until the number of times the pixel point generates an event is greater than the event filtering threshold);

[0183] 8. Put the calculation results into historical data;

[0184] Taking the change of pixel value as 0-255-255-0 as an example, the calculation result may be that the pixel value at 255 of the second frame is different from that of the first frame, and the second frame and its pixel value 255 may be stored. If the pixel value at the fourth frame is the same as that of the first frame, the fourth frame and its pixel value 0 may be stored. That is, the calculation result may include the data frame and the corresponding pixel value when the event is about to occur, and the data frame and the corresponding pixel value when the event occurs.

[0185] 9. Then clustering is performed based on the points with the same frequency on the current frame, and the areas with the same frequency are considered to be the same object (i.e., based on the event frequency corresponding to each pixel point in the current frame, the pixel points are clustered to obtain the object included in the current frame);

[0186] 10. After clustering is completed, the center point of the area (the same object) and the average frequency are output (that is, for each object obtained by clustering, the object frequency of the object is determined based on the event frequency of the pixel points corresponding to the object).

[0187] In this embodiment, the pixels of the current frame and the pixels of the reference frame are analyzed to determine whether an event occurs, and the object frequency is determined by event frequency clustering, making the method of determining the object frequency simpler and faster.

[0188] Embodiment 5

[0189] Figure 5 This is a schematic diagram of the structure of an object frequency determination device provided in Embodiment 5 of the present invention. Figure 5 As shown, the device comprises:

[0190] An acquisition module 410 is used to acquire a current frame;

[0191] An analysis module 420 is used to analyze, for each pixel in the current frame, the pixel in the current frame and the reference pixel in the reference frame to determine whether the pixel generates an event, and the position of the reference pixel in the reference frame is the same as the position of the pixel in the current frame;

[0192] A first determination module 430 is used to determine the event frequency of the event based on the time taken for the event to occur when the event occurs at the pixel point;

[0193] A clustering module 440, configured to cluster the pixels based on the event frequency corresponding to each pixel in the current frame to obtain the objects included in the current frame;

[0194] The second determination module 450 is used to determine the object frequency of each object obtained by clustering based on the event frequency of the pixel points corresponding to the object.

[0195] Optionally, the acquisition module 410 is specifically configured to:

[0196] Acquire an initial data stream, wherein the initial data stream includes multiple frames of grayscale images;

[0197] After filtering the gray part of each grayscale image frame in the initial data stream, the target data stream is obtained;

[0198] A current frame is obtained from the target data stream.

[0199] Optionally, the analysis module 420 includes:

[0200] A first selection unit, configured to use, when the current frame is the first frame, the nth frame after the current frame in the target data stream as a first reference frame;

[0201] a comparing unit, configured to compare a pixel value of a reference pixel point in the first reference frame with a pixel value of the pixel point in the current frame to determine whether an event is to be generated;

[0202] a second selection unit, configured to, if an event is to be generated, use the mth frame after the first reference frame as a second reference frame, and compare the pixel value of the reference pixel point in the second reference frame with the pixel value of the pixel point in the current frame to determine whether an event is to be generated;

[0203] a third selection unit, configured to, if an event is not about to occur, use the kth frame after the first reference frame as a new first reference frame, and continue to determine whether an event is about to occur until an event is about to occur;

[0204] Wherein, m, n and k are positive integers.

[0205] Optionally, a comparison unit is used to:

[0206] If the pixel value of the reference pixel in the first reference frame is different from the pixel value of the pixel in the current frame, it is determined that an event will occur; otherwise, it is determined that no event will occur.

[0207] Optionally, the second selection unit is specifically used to:

[0208] If the pixel value of the reference pixel in the second reference frame is the same as the pixel value of the pixel in the current frame, it is determined that an event has occurred; otherwise, no event has occurred.

[0209] Optionally, the analysis module 420 further includes:

[0210] A fourth selection unit, configured to select a reference frame from reference events of historical data when the current frame is not the first frame;

[0211] A comparison unit is used to determine whether an event occurs at the pixel point based on a comparison result between a pixel value of the reference pixel point of the reference frame and a pixel value of the pixel point of the current frame.

[0212] Optionally, the comparison unit is specifically configured to determine whether the pixel point generates an event based on a comparison result of a pixel value of a reference pixel point of the reference frame and a pixel value of the pixel point of the current frame when the reference frame is a frame having the largest frame interval with the current frame in the reference event, including:

[0213] If the pixel value of the reference pixel of the reference frame is the same as the pixel value of the pixel of the current frame, it is determined that the pixel generates an event; otherwise, continue to select the next frame after the current frame from the target data stream as the new current frame, and continue to perform the operation of determining whether the pixel generates an event based on the comparison result of the pixel value of the reference pixel of the reference frame with the pixel value of the pixel of the current frame until an event is generated.

[0214] Optionally, the first determining module 430 is specifically configured to:

[0215] Determine the number of frames used to determine whether an event occurs;

[0216] Determine the time taken for the event camera to output the number of frames used;

[0217] The inverse of the elapsed time is determined as the event frequency of the event.

[0218] Optionally, the second determination module 450 is specifically used for determining that, when a frequency range formed by event frequencies of pixels included in the same object is within a set range, the object frequency corresponding to each object is an average of the event frequencies of the pixels corresponding to the object.

[0219] Optionally, the device further includes a smoothing processing module, configured to:

[0220] When the used time is less than the set interval threshold, the event frequency of the event is smoothed according to the last used time of the last event and the used time of the event.

[0221] Optionally, the smoothing processing module is specifically used to:

[0222] Determining a weight coefficient based on a ratio of the time taken for the event to the last time taken for the last event;

[0223] Based on the weight coefficient, the event frequencies of the events are weighted and summed to obtain updated event frequencies.

[0224] Optionally, the first determining module 430 is specifically configured to:

[0225] In the case where an event is generated at the pixel point, if the number of times the pixel point generates events is greater than the event filtering threshold, the event frequency of the event is determined based on the time taken for the event to be generated; otherwise, continue to determine whether the pixel point generates the next event until the number of times the pixel point generates events is greater than the event filtering threshold.

[0226] The object frequency determination device provided in the embodiment of the present invention can execute the object frequency determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0227] Embodiment 6

[0228] Figure 6A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0229] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by the at least one processor 11, and the computer program is executed by the at least one processor 11 so that the at least one processor 11 can execute the method provided by the present invention.

[0230] The processor 11 can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 12 or a computer program loaded from a storage unit 18 into a random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0231] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0232] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the object frequency determination method.

[0233] In some embodiments, the method object frequency determination may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the object frequency determination method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the object frequency determination method in any other suitable manner (e.g., by means of firmware).

[0234] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0235] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0236] In the context of the present invention, a computer-readable storage medium stores computer instructions, which are used to implement a method for determining the frequency of an object provided by the present invention when the processor executes the computer-readable storage medium. The computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0237] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD monitor)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user may provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0238] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN), blockchain network, and the Internet.

[0239] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0240] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0241] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining an object frequency, characterized in that: include: Get the current frame; For each pixel in the current frame, analyzing the pixel in the current frame and a reference pixel in a reference frame to determine whether the pixel generates an event, wherein a position of the reference pixel in the reference frame is the same as a position of the pixel in the current frame; When an event occurs at the pixel point, determining an event frequency of the event based on the time taken for the event to occur; Clustering the pixels based on the event frequency corresponding to each pixel in the current frame to obtain the objects included in the current frame; For each object obtained by clustering, the object frequency of the object is determined based on the event frequency of the pixel points corresponding to the object.

2. The method according to claim 1, characterized in that The obtaining of the current frame comprises: Acquire an initial data stream, wherein the initial data stream includes multiple frames of grayscale images; After filtering the gray part of each grayscale image frame in the initial data stream, the target data stream is obtained; A current frame is obtained from the target data stream.

3. The method according to claim 1, characterized in that In the case where the current frame is the first frame, analyzing the pixel point of the current frame and the reference pixel point of the reference frame to determine whether the pixel point generates an event includes: Taking the nth frame after the current frame in the target data stream as the first reference frame; Compare the pixel value of the reference pixel in the first reference frame with the pixel value of the pixel in the current frame to determine whether an event is about to occur; If an event is to be generated, the mth frame after the first reference frame is used as a second reference frame, and the pixel value of the reference pixel point in the second reference frame is compared with the pixel value of the pixel point in the current frame to determine whether an event is generated; If an event is not about to occur, taking the kth frame after the first reference frame as a new first reference frame, and continuing to determine whether an event is about to occur until an event is about to occur; Wherein, m, n and k are positive integers.

4. The method according to claim 3, characterized in that Comparing the pixel value of the reference pixel in the first reference frame with the pixel value of the pixel in the current frame to determine whether an event is about to occur, comprises: If the pixel value of the reference pixel in the first reference frame is different from the pixel value of the pixel in the current frame, it is determined that an event will occur; otherwise, it is determined that no event will occur.

5. The method according to claim 3, characterized in that: The comparing the pixel value of the reference pixel point in the second reference frame with the pixel value of the pixel point in the current frame to determine whether an event is generated includes: If the pixel value of the reference pixel in the second reference frame is the same as the pixel value of the pixel in the current frame, it is determined that an event has occurred; otherwise, no event has occurred.

6. The method according to claim 1, characterized in that In the case that the current frame is not the first frame, analyzing the pixel point and the reference pixel point of the reference frame to determine whether the pixel point generates an event includes: Select a reference frame from the reference events of the historical data; Based on a comparison result between a pixel value of a reference pixel of the reference frame and a pixel value of the pixel of the current frame, it is determined whether the pixel generates an event.

7. The method according to claim 6, characterized in that When the reference frame is a frame having the largest frame interval with the current frame in the reference event, the comparing result of the pixel value of the reference pixel of the reference frame with the pixel value of the pixel of the current frame to determine whether the pixel generates an event comprises: If the pixel value of the reference pixel of the reference frame is the same as the pixel value of the pixel of the current frame, it is determined that the pixel generates an event; otherwise, continue to select the next frame after the current frame from the target data stream as the new current frame, and continue to perform the operation of determining whether the pixel generates an event based on the comparison result of the pixel value of the reference pixel of the reference frame with the pixel value of the pixel of the current frame until an event is generated.

8. The method according to claim 1, characterized in that The determining the event frequency of the event based on the time taken for the event to occur comprises: Determine the number of frames used to determine whether an event occurs; Determine the time taken for the event camera to output the number of frames used; The inverse of the elapsed time is determined as the event frequency of the event.

9. The method according to claim 1, characterized in that: The frequency range formed by the event frequencies of the pixels included in the same object is within the set range, and the object frequency corresponding to each object is the average of the event frequencies of the pixels corresponding to the object.

10. The method according to claim 1, characterized in that Also includes: When the used time is less than the set interval threshold, the event frequency of the event is smoothed according to the last used time of the last event and the used time of the event.

11. The method according to claim 10, characterized in that The smoothing of the event frequency of the event according to the last time taken by the last event and the time taken by the event includes: Determining a weight coefficient based on a ratio of the time taken for the event to the last time taken for the last event; Based on the weight coefficient, the event frequencies of the events are weighted and summed to obtain updated event frequencies.

12. The method according to claim 1, characterized in that In the case where an event is generated at the pixel point, determining the event frequency of the event based on the time taken for the event to be generated includes: In the case where an event is generated at the pixel point, if the number of times the pixel point generates events is greater than the event filtering threshold, the event frequency of the event is determined based on the time taken for the event to be generated; otherwise, continue to determine whether the pixel point generates the next event until the number of times the pixel point generates events is greater than the event filtering threshold.

13. An object frequency determination device, characterized in that: include: Acquisition module, used to obtain the current frame; An analysis module, configured to analyze, for each pixel in the current frame, the pixel in the current frame and the reference pixel in the reference frame to determine whether the pixel generates an event, and the position of the reference pixel in the reference frame is the same as the position of the pixel in the current frame; A first determination module is used to determine the event frequency of the event based on the time taken for the event to occur when the event occurs at the pixel point; A clustering module, used for clustering the pixels in the current frame based on the event frequency corresponding to each pixel in the current frame to obtain the objects included in the current frame; The second determination module is used to determine the object frequency of each object obtained by clustering based on the event frequency of the pixel points corresponding to the object.

14. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the object frequency determination method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the object frequency determination method according to any one of claims 1 to 2 when executed.

Images