A light field imaging method with cross-scale adaptive mapping
Through the light field imaging method of cross-scale adaptive mapping, the target moving objects in video surveillance are screened out, which solves the problem of redundant data and information loss in high-definition video acquisition and improves data processing efficiency.
Patent Information
- Application Number
- CN202110019681.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-07
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-01-07
AI Technical Summary
In video surveillance, the acquisition and storage of high-definition video sequences puts high requirements on hardware computing power and algorithms, resulting in continuous acquisition in a static environment to generate a large amount of redundant data, and the high sensitivity of the motion trigger monitoring system leads to information loss and analysis Slow problem.
The light field imaging method with cross-scale adaptive mapping is adopted to obtain data from the light field camera and event camera, determine the motion significance index of dynamic objects, filter out the target moving objects, and reduce the acquisition and processing of unnecessary data.
It effectively reduces redundant data collection and processing, improves data analysis speed, avoids information loss, and achieves accurate identification and efficient processing of target objects.
Smart Images

Figure CN112884805B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to image processing technology, and in particular to a light field imaging method for cross-scale adaptive mapping. Background Art
[0002] With the improvement of the degree of social informatization, the collection, analysis and storage of big data have gradually become bottlenecks that restrict its further development. In the field of video surveillance, the performance indicators such as camera resolution and frame rate have gradually improved. However, high-definition video sequences have put forward higher requirements for storage and subsequent analysis in terms of hardware computing power and efficient algorithms. In many monitoring scenarios, the monitored objects or environments are static and stable most of the time, but high-definition monitoring cameras are still shooting continuously, generating a large amount of data to be saved in real time. Although these data are huge in volume, the amount of effective information contained in them is negligible, and even has no value for practical applications. In this case, even if the collection and storage rates are increased to a higher level, it actually does not work. In order to avoid such huge redundant information collection, motion-triggered monitoring camera systems are increasingly used in production and life. In this system, the surveillance camera generally does not collect data. The collection function is only triggered when a moving object appears in the field of view; when the movement stops and the picture is stable for a period of time, the collection stops.
[0003] Although motion-triggered monitoring systems can avoid continuous invalid data collection when the image is still for a long time, and to a certain extent alleviate the huge pressure on data storage, in order to ensure that information is not missed, the sensitivity of motion triggering is generally relatively high. For example, when you want the monitoring system to trigger the collection when pedestrians pass by, but due to its high sensitivity, it starts collecting when birds fly by. When subsequent data analysis is performed after data collection is completed, due to problems such as high image resolution and single information source, it causes defects such as slow analysis and information loss. Summary of the invention
[0004] The present invention provides a cross-scale adaptive mapping light field imaging method and device to achieve accurate recognition of target moving objects and determination of target images in collected images, thereby improving the processing efficiency of subsequent data.
[0005] In a first aspect, an embodiment of the present invention provides a light field imaging method for cross-scale adaptive mapping, the method comprising:
[0006] Acquire a light field image captured by a light field camera and a grayscale image captured by an event camera, and determine a current regional distribution image containing at least one dynamic object and a motion speed corresponding to each of the dynamic objects according to the light field image and the grayscale image;
[0007] Determining a motion saliency index according to the light field image, the current regional distribution image and the motion speed corresponding to each of the dynamic objects;
[0008] Screening each of the dynamic objects according to the motion significance index to obtain a target motion object;
[0009] A target image is determined according to the target moving object.
[0010] In a second aspect, an embodiment of the present invention further provides a light field imaging device for cross-scale adaptive mapping, the device comprising:
[0011] an acquisition module, configured to acquire a light field image acquired by a light field camera and a grayscale image acquired by an event camera, and determine a current regional distribution image containing at least one dynamic object and a motion speed corresponding to each of the dynamic objects according to the light field image and the grayscale image;
[0012] An index determination module, used to determine a motion significance index according to the light field image, the current regional distribution image and the motion speed corresponding to each of the dynamic objects;
[0013] An object screening module, used for screening each of the dynamic objects according to the motion significance index to obtain a target motion object;
[0014] The image determination module is used to determine a target image according to the target moving object.
[0015] The embodiment of the present invention provides a light field imaging method and device for cross-scale adaptive mapping, which obtains a light field image collected by a light field camera and a grayscale image collected by an event camera, and determines a current regional distribution image containing at least one dynamic object and a motion speed corresponding to each dynamic object according to the light field image and the grayscale image; determines a motion significance index according to the light field image, the current regional distribution image and the motion speed corresponding to each dynamic object; and screens each dynamic object according to the motion significance index to obtain a target moving object, thereby solving the problem of frequent, large-scale image collection and processing due to slight changes when collecting images of a monitoring area. By analyzing the image, each dynamic object in the image is determined, and then the motion significance index is determined. According to the motion significance index, each dynamic object is screened to obtain the target moving object, which can effectively filter out dynamic objects that are not needed or expected to be paid attention to in the image. When the target moving object is not included in the image, the image does not need to be processed, which effectively reduces the amount of data processing, reduces the difficulty of subsequent data processing, and improves the speed of data analysis. By identifying the target moving object for subsequent data processing, the problem of information loss caused by excessive data volume can be effectively avoided. The target moving object can be intuitively observed through the target image, which is convenient for subsequent data processing or execution of other operations. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 is a flow chart of a light field imaging method for cross-scale adaptive mapping in Embodiment 1 of the present invention;
[0017] Figure 2 is a flow chart of a light field imaging method for cross-scale adaptive mapping in Embodiment 2 of the present invention;
[0018] Figure 3 This is an example diagram of an implementation of determining a motion significance index in Embodiment 2 of the present invention;
[0019] Figure 4 It is a structural schematic diagram of a light field imaging device for cross-scale adaptive mapping in Embodiment 3 of the present invention. DETAILED DESCRIPTION
[0020] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0021] Embodiment 1
[0022] Figure 1 This is a flow chart of a light field imaging method for cross-scale adaptive mapping provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of screening target moving objects in an image and determining a target image. The method can be performed by a light field imaging device for cross-scale adaptive mapping, and specifically includes the following steps:
[0023] Step S110: acquiring a light field image captured by a light field camera and a grayscale image captured by an event camera, and determining a current regional distribution image containing at least one dynamic object and a motion speed corresponding to each dynamic object according to the light field image and the grayscale image.
[0024] In this embodiment, the light field camera does not need to select the focus before taking a photo. You can take a photo first. The camera captures a large amount of light information and selects the focus. It is more flexible when taking photos. The camera records all optical information within the focal length range when taking photos, without sacrificing depth of field and image clarity. The event camera does not capture images at a fixed frame rate, but outputs asynchronous signals (including events, positions, and signs of brightness changes) by measuring the brightness changes of each pixel. The output of the event camera is in the form of an event data stream. Each event E iIt is expressed in the form of (x, y, t, event). (x, y) represents the position of the image sensor, t represents the time of occurrence, and event represents the category of the event. In common event cameras, event represents the change in light intensity detected by the photodiode, 1 represents an increase in light intensity, and -1 represents a decrease. That is, when the brightness value of a pixel changes, the event camera will return an event in the above format. The light field image can be understood as the image collected by the light field camera when it is working; the grayscale image can be understood as the image composed of event information collected by the event camera when one or more events are triggered; the dynamic object can be understood as the object that moves when the image collected at the current moment is compared with the image collected at the previous moment; the current area distribution image can be understood as an image that contains dynamic objects and can indicate the location of the dynamic objects.
[0025] After receiving the instruction to start working, the light field camera collects light field images. After capturing the movement of an object or subject, the event camera feeds back the change information in the form of an event and forms a grayscale image. The light field image and the grayscale image are processed by an algorithm or a deep learning network model to obtain a current regional distribution image. The current regional distribution image includes one or more dynamic objects. When the light field image and the grayscale image are processed to obtain the current regional distribution image, the movement speed of each dynamic object is also obtained.
[0026] Step S120: determining a motion saliency index according to the light field image, the current regional distribution image and the motion speed corresponding to each dynamic object.
[0027] In this embodiment, the motion significance index can be understood as data indicating whether each pixel in the image moves or the degree of motion. The motion significance index is obtained by comprehensive analysis of multiple aspects, and the motion significance index is determined by speed, depth, size, or whether a new dynamic object is added.
[0028] The speed of each pixel in the current regional distribution image can be determined by the current regional distribution image and the corresponding motion speed of each dynamic object; by comparing and analyzing the current regional distribution image with the image collected last time, it is determined whether there is a newly added dynamic object; by processing the light field image, the depth of each pixel in the image is obtained, and then the depth of each dynamic object is obtained; the actual size of the dynamic object is determined by the depth of the dynamic object, the parameters of the light field camera and the size of the dynamic object mapped on the light field camera, and the motion significance index is comprehensively determined based on one or more of the above factors.
[0029] Step S130: Screen each dynamic object according to the motion significance index to obtain a target motion object.
[0030] In this embodiment, the target moving object can be understood as the moving object that needs to be paid attention to in the captured image. For example, when monitoring a road section, the road section is photographed. The people or cars that appear in the photographed area are the target moving objects that need to be paid attention to, but the flying birds and passing kittens do not need to be paid attention to. Therefore, the dynamic objects collected in the image may include multiple dynamic objects such as birds, kittens, people, and cars. The target moving objects that need to be paid attention to are screened from multiple dynamic objects. The motion significance index obtained by integrating multiple influencing factors is used as the basis for dynamic object screening. The target moving object is determined by judging whether the motion significance index meets the requirements. The target moving object may be 0, 1 or more.
[0031] Step S140: determine a target image according to the target moving object.
[0032] In this embodiment, the target image can be understood as the image finally formed after the target moving object is screened. After the target moving object is determined, the pixel values of all pixels in the single connected area formed by the target moving object are recorded as 1, and the remaining pixels are all backgrounds, and the pixel values are recorded as 0 to form the target image. When there is a target moving object, the obtained target image can contain one or more target moving objects, and the target moving object can be intuitively seen from the target image; when there is no target moving object, the pixel values of all pixels in the target image are 0. The target moving object can be intuitively observed through the target image, which can be used for subsequent shooting of the target moving object. For example, when it is determined that the target moving object is a car, the camera (without limiting the camera type) can be controlled to shoot the car more clearly and in a targeted manner, such as adjusting the camera focal length according to the depth (distance) of the car reflected in the target image, and shooting the vehicle image.
[0033] The embodiment of the present invention provides a light field imaging method for cross-scale adaptive mapping, which obtains a light field image captured by a light field camera and a grayscale image captured by an event camera, and determines a current regional distribution image containing at least one dynamic object and a motion speed corresponding to each dynamic object according to the light field image and the grayscale image; determines a motion significance index according to the light field image, the current regional distribution image and the motion speed corresponding to each dynamic object; screens each dynamic object according to the motion significance index to obtain a target motion object, and determines a target image according to the target motion object, thereby solving the problem of frequent and large-scale image acquisition due to slight changes when acquiring images of a monitoring area. Since the pixel levels of the grayscale image and the light field image are similar, the target image is obtained. The difference is large. By analyzing the light field image and the grayscale image, the dynamic objects in the image are determined, and cross-scale image analysis is realized. The dynamic objects are screened according to the motion significance index to obtain the target moving object, which can effectively filter out the dynamic objects that are not needed or expected to be paid attention to in the image. When the image does not contain the target moving object, there is no need to process the image, which effectively reduces the amount of data processing, reduces the difficulty of subsequent data processing, and improves the speed of data analysis. By identifying the target moving object for subsequent data processing, the problem of information loss caused by excessive data volume can be effectively avoided; the target moving object can be intuitively observed through the target image, which is convenient for subsequent data processing or other operations.
[0034] Embodiment 2
[0035] Figure 2 This is a flow chart of a light field imaging method for cross-scale adaptive mapping provided in Embodiment 2 of the present invention. The technical solution of this embodiment is further refined on the basis of the above technical solution, and specifically mainly includes the following steps:
[0036] Step S210: receiving a trigger event sent by an event camera and adding 1 to the trigger count of the trigger event.
[0037] In this embodiment, the trigger event can be understood as an event that is triggered, generated and sent when the event camera detects the movement of an object. The trigger count can be understood as the number of occurrences of the trigger event.
[0038] Event cameras do not have the array output limitations of conventional image sensors. When each photodiode detects a change in intensity, a trigger event is generated immediately and output to a computer or other processing device through a USB bus structure. Due to the sensitivity of photodiodes, the generation of a trigger event does not necessarily mean that the object at the corresponding position in the scene has shifted. It may also be imaging noise caused by subtle changes in the environment. Such effects can be eliminated by software in the computer, or the event camera can filter such trigger events by itself to reduce the number of trigger events sent to a computer or other data processing terminal to avoid affecting data processing. The computer terminal device that performs data processing receives the trigger event sent by the event camera, and then accumulates the number of trigger events by 1. The number of triggers can be accumulated by a counter. After each trigger event is received, the counter count is controlled to accumulate by 1.
[0039] Step S220: When it is detected that the number of triggering times is greater than a preset event threshold, the light field camera is controlled to collect a light field image.
[0040] In this embodiment, the preset event threshold can be understood as a preset total threshold value, which is used to determine whether the number of occurrences of the trigger event meets the requirement. When the total number of occurrences of the trigger event exceeds the preset event threshold, that is, the number of triggers is greater than the preset event threshold, it is determined that an object has moved in the environment at this time, and the light field camera is controlled to work and collect light field images.
[0041] After this step, the counted number of trigger events is cleared, and the number of received trigger events is recounted to stimulate the acquisition of light field images cyclically. The accumulation of the number of triggers represents the severity of the changes in the scene. Through this acquisition method, the global image of the scene is only acquired when the scene changes strongly enough. The advantages of this acquisition method are: it is free from the limitation of traditional cameras to acquire at fixed time intervals, and can adaptively control the acquisition according to the motion conditions in the scene, which largely avoids the acquisition of static redundant information and reduces the cost of subsequent data storage and analysis.
[0042] The event camera can also trigger the light field camera to start working. When the number of triggers is greater than the preset event threshold, the event camera is controlled to send a control instruction to the light field camera to control the light field camera to collect light field images. Alternatively, the event camera counts the number of triggers by itself, and the accumulation method is consistent with the above method. When the number of triggers is greater than the preset event threshold, the light field camera is controlled to collect light field images. The event camera and the light field camera can be placed in a parallel fixed manner.
[0043] Step S230: determine a grayscale image according to the event information of each triggering event.
[0044] In this embodiment, the event information can be understood as the data information of each trigger event, for example, the location (horizontal and vertical coordinates) of occurrence, the time of occurrence, and the category of the event (i.e., light intensity increases or decreases). Since the event information of each trigger event is different, each trigger event indicates what kind of changes have occurred at a certain moment and a certain position. The event information of multiple trigger events is superimposed to obtain a grayscale image. For example, the event information is: (x1, y1, t1, 1), (x1, y1, t2, 1), (x2, y2, t3, -1). The above event information indicates that at (x1, y1), the light intensity increases twice, and at (x2, y2), the light intensity decreases once. The trigger events are superimposed according to their respective locations and densities (density refers to the location or area captured by a radio and television diode) to form a frame of grayscale image.
[0045] Step S240: Acquire a light field image captured by the light field camera and a grayscale image captured by the event camera.
[0046] Step S250: determining a pan-focus image based on the light field image in combination with a preset algorithm.
[0047] In this embodiment, the preset algorithm can be an image processing algorithm provided by the manufacturer of the light field camera, or a model of a deep neural network structure such as EPI-net, which can perform analytical operations on light field depth data; a pan-focus image refers to a clear image of all images within a certain range of the captured image. Compared with traditional cameras, light field cameras can collect four-dimensional light field information, that is, on the basis of the original intensity distribution, the direction information of the light is added. The light field image is processed by a preset algorithm to obtain a pan-focus image, which realizes a simple method to obtain a large depth of field pan-focus image of the current scene.
[0048] Step S260: Combine the pan-focus image and the grayscale image with a predetermined neural network model to determine a current area distribution image containing at least one dynamic object and a motion speed corresponding to each dynamic object.
[0049] In this embodiment, the predetermined neural network model can be a U-net full convolutional network; the neural network model is pre-trained, and then the pan-focus image and the grayscale image are used as the input of the model and input into the neural network model to obtain the current regional distribution image and the motion speed of each dynamic object output by the model. The current regional distribution image is a binary mask, 0 indicates that this position is the background, and 1 indicates that this position is a dynamic object.
[0050] Step S270: determining at least one motion parameter matrix according to the light field image, the current regional distribution image and the motion speeds corresponding to each dynamic object.
[0051] In this embodiment, the motion parameter matrix can be understood as a distribution matrix formed according to different parameters of the dynamic object during the motion process. Each element in the matrix represents a pixel point, and the value on each element represents the pixel value corresponding to this pixel point. The motion parameter matrix can be one or more of the following matrices: a velocity distribution matrix, a depth distribution matrix, a size distribution matrix, or a newly added object matrix. Through the current regional distribution image, it can be determined whether there is a dynamic object at each pixel in the image, and the velocity distribution matrix is determined according to the motion speed of the dynamic object. The depth of each pixel is determined by the light field image, and then the depth of the dynamic object is determined, and then the depth distribution matrix is determined; the actual size of the dynamic object is determined, and the size distribution matrix is determined according to the actual size of the dynamic object; it is determined whether there is a newly added object in the dynamic object, and the newly added object matrix is determined.
[0052] Furthermore, when the motion parameter matrix is a velocity distribution matrix, correspondingly, determining the velocity distribution matrix can be implemented in the following manner:
[0053] According to the current regional distribution image, the regional pixel point set where each dynamic object is located is determined, and the regional pixel point set includes at least one regional pixel point; the movement speed of each dynamic object is determined as the target speed of each regional pixel point in the corresponding regional pixel point set, and a speed distribution matrix is formed according to the positions of all regional pixel points and the corresponding target speeds.
[0054] In this embodiment, the speed distribution matrix can be understood as a matrix determined by the motion speed of the dynamic object. The regional pixel point can be understood as the pixel point where the dynamic object is located in the image, and the position of each dynamic object in the image is composed of one or more pixel points; the regional pixel point set can be understood as a set composed of pixel points corresponding to each dynamic object; and one regional pixel point set corresponds to one dynamic object. The target speed can be understood as the predicted speed of the regional pixel point.
[0055] The current regional distribution image contains a dynamic object, and the current regional distribution image is composed of multiple pixels. Each regional pixel where the dynamic object is located can be determined from the current regional distribution image, and all regional pixels corresponding to a dynamic object constitute a regional pixel set. For each dynamic object, the movement speed of the dynamic object is used as the target speed of each regional pixel in the regional pixel set corresponding to the dynamic object, thereby obtaining the target speed of all regional pixels, and forming a speed distribution matrix according to the positions of all regional pixels and the corresponding target speeds. The target speed of pixels not covered by the dynamic object is 0. The speed distribution matrix is represented by a two-dimensional matrix VelocityMask, which has the same resolution as the current regional distribution image of the dynamic object, and the grayscale value of the corresponding position represents the magnitude of the relative speed. When forming the speed distribution matrix, the movement speed of each dynamic object can be normalized. For example, the movement speed of dynamic object 1 is 20 pixels / s, and the movement speed of dynamic object 2 is 40 pixels / s. After normalization, the movement speed of dynamic object 1 is 0.5 pixels / s, and the movement speed of dynamic object 2 is 1 pixel / s.
[0056] Exemplarily, the velocity distribution matrix is:
[0057]
[0058] Furthermore, when the motion parameter matrix is a depth distribution matrix, correspondingly, determining the depth distribution matrix may be implemented in the following manner:
[0059] A depth map is determined based on the light field image combined with a preset algorithm; the depth of each pixel point in each area pixel point set is determined based on the depth map, and the average value of each depth is determined as the object depth of the corresponding dynamic object; a depth distribution matrix is formed based on the positions of all area pixel points and the corresponding object depths.
[0060] In this embodiment, the depth map can be understood as an image reflecting the distance between the data corresponding to each pixel and the event camera, and the depth also refers to the distance. The object depth is the distance between the dynamic object and the event camera. The depth distribution matrix can be understood as a matrix determined by the object depth of the dynamic object.
[0061] The light field image is processed by a preset algorithm to obtain a depth map. Each pixel in the depth map has a corresponding depth. The depths of the pixels in each region in a regional pixel set are averaged to obtain the object depth of the dynamic object corresponding to this regional pixel set. Then, the regional pixels in the regional pixel set corresponding to this dynamic object correspond to the object depth, thereby obtaining the depth of all regional pixels. A depth distribution matrix is formed according to the positions of all regional pixels and the corresponding object depths. The depth of pixels not covered by dynamic objects is 0. The depth distribution matrix is represented by a two-dimensional matrix DepthMask. When forming the depth distribution matrix, the object depth of each dynamic object can also be normalized.
[0062] Furthermore, when the motion parameter matrix is a size distribution matrix, correspondingly, determining the size distribution matrix may be implemented in the following manner:
[0063] Determine the number of horizontal pixels corresponding to each dynamic object on the image sensor of the light field camera; determine the actual size of each dynamic object according to the light field camera parameters, the object depth and the number of horizontal pixels corresponding to each dynamic object combined with a predetermined formula; and form a size distribution matrix according to the positions of all area pixel points and the corresponding actual sizes.
[0064] In this embodiment, the size distribution matrix can be understood as a matrix determined by the actual size of the dynamic object. The number of horizontal pixels can be understood as the number of pixels occupied by the dynamic object in the set horizontal direction when the dynamic object is imaged on the image sensor of the light field camera. From the image point of view, that is, how many pixels are occupied by the widest part of the dynamic object in the horizontal direction (width). The light field camera parameters can be the pixel size of the light field camera image sensor and the focal length of the lens.
[0065] The light field camera parameters are already determined when shooting. Usually, when monitoring a certain area, the light field camera is set at a certain position. At this time, the light field camera parameters have been determined. If they are not changed, the light field camera parameters are fixed and unchanged. After shooting the light field image, the light field image can be used to determine the number of horizontal pixels corresponding to each dynamic object on the image sensor of the light field camera. The light field camera parameters, the object depth and the number of horizontal pixels corresponding to each dynamic object are brought into the predetermined formula. The actual size of each dynamic object can be obtained by calculation, and then the regional pixels in the regional pixel set corresponding to each dynamic object are matched with the actual size, thereby obtaining the size corresponding to all regional pixels. A size distribution matrix is formed according to the positions of all regional pixels and the corresponding actual sizes. The actual size of pixels not covered by dynamic objects is 0. The size distribution matrix is represented by a two-dimensional matrix ScaleMask. When forming the size distribution matrix, the actual size of each dynamic object can also be normalized.
[0066] Exemplarily, an embodiment of the present invention provides an expression of a predetermined formula:
[0067]
[0068] Where, f is the focal length of the lens; N in is the number of horizontal pixels; PixelSize is the pixel size of the light field camera sensor; Depth is the object depth; Scale is the actual size. The method of estimating the actual size by the object depth is to use the similarity principle of the camera's internal triangle.
[0069] Furthermore, when the motion parameter matrix is a newly added object matrix, correspondingly, determining the newly added object matrix may be implemented in the following manner:
[0070] The newly added object image is determined according to the current regional distribution image and the previous regional distribution image; and the newly added object matrix is determined according to the position and pixel value of each pixel point in the newly added object image.
[0071] In this embodiment, the newly added object matrix can be understood as a matrix formed according to the newly added dynamic objects; the previous regional distribution image can be understood as the regional distribution image collected last time; the newly added object image can be understood as an image containing the newly added dynamic objects.
[0072] The last regional distribution image and the current regional distribution image collected last time are respectively subjected to feature extraction to determine whether there are newly added dynamic objects (i.e., whether new connected areas have appeared). By comparing the current regional distribution image and the previous regional distribution image, the newly added object image is determined. The shape of the newly added object image is the same as the current regional distribution image. Only the pixel value of the area corresponding to the newly appeared dynamic object (the pixel value of the connected area) is 1, and the pixel values of the remaining areas (including the connected areas of the previously existing dynamic objects and the background area) are 0. The newly added object matrix is determined according to the position and pixel value of each pixel point in the newly added object image.
[0073] Step S280: Determine the sum of the products of each motion parameter matrix and the corresponding weight as the motion significance index.
[0074] A weight is set for each different type of motion parameter matrix, each motion parameter matrix is multiplied by the corresponding weight, and then the products are added together to obtain a matrix that is a motion saliency index.
[0075] The motion saliency index comprehensively considers the motion speed, object depth (distance), actual size, and whether it is a newly added object of each detected dynamic object to determine the degree of its attention, that is, the motion saliency index is determined by one or more of the following: the motion speed of each dynamic object, the object depth of each dynamic object, the actual size of each dynamic object, and the newly added object image. Exemplarily, the embodiment of the present invention provides a formula for calculating the motion saliency index:
[0076] SignificanceMask=
[0077] w 1 *VelocityMask+w 2 *DepthMask+w 3 *ScaleMask+w 4 *NewComerMask
[0078] where w 1 +w 2 +w 3 +w 4 =1;
[0079] In the above formula, w 1 、w 2 、w 3 、w 4 represents the weight of each indicator, that is, their importance to the final adaptive judgment result. For example, if you want to focus on fast dynamic objects, you only need to increase w 1 If you want to focus on the dynamic object at the farthest distance, you need to increase the proportion w 2 If you want to focus on the dynamic object with the largest actual size, you need to increase the ratio w 3 ; If you want to focus on the newly added dynamic objects, you need to increase the ratio w 4 .
[0080] The user can customize the weight according to the actual monitoring task. The system can adaptively calculate the motion significance index after each dynamic object is collected. The motion significance index is expressed in the form of pixel values at the corresponding position. Each dynamic object corresponds to (or constitutes) a connected area. The pixel values of the pixels in each connected area are the same. The motion significance index in the embodiment of the present application includes all the pixels in an image. Each dynamic object can also be segmented and judged separately. The principle is the same, except that the matrix size corresponding to the motion significance index is different. By adjusting the weights corresponding to different motion parameter matrices, different motion significance indexes can be obtained to achieve different focuses on dynamic objects.
[0081] Step S290: Determine the dynamic object corresponding to each indicator data greater than the preset output threshold as the target moving object.
[0082] In this embodiment, the preset output threshold can be understood as a preset output value measurement standard, which is used to determine whether the pixel value of each pixel point meets the output standard, that is, whether it can be output. The object composed of the pixels that can be output is the target motion object. The motion significance index is actually a matrix, and the data on each element in the matrix is the indicator data. The motion significance index includes at least one indicator data. The preset output threshold can be set in the form of a matrix, or it can be just a numerical value. When it is set in the form of a matrix, the number of rows and columns of the matrix is the same as the number of rows and columns of the motion significance index, and the numerical value of each element in the matrix is the same. Each indicator data in the motion significance index is compared with the element at the corresponding position respectively; when it is set as a numerical value, it is directly compared with each indicator data in the motion significance index. Determine each indicator data greater than the preset output threshold, and one or more dynamic objects can be correspondingly determined according to all indicator data greater than the preset output threshold, and the obtained dynamic object is determined as the target motion object.
[0083] Here is an example of how to determine the target moving object:
[0084] The motion significance index is:
[0085]
[0086] Taking 0.5 as an example of the preset output threshold, after comparison, only the position with the element of 0.8 in the matrix of the motion saliency index can be output, that is, only the index data with a value of 0.8 meets the conditions, and all the index data equal to 0.8 constitute a square-shaped dynamic object, which is the target dynamic object; while the index data with a value of 0.3 does not meet the conditions, although the T-shaped object it constitutes is also a dynamic object, this dynamic object is an object that does not want to be paid attention to, and is not output as a target motion object. If the motion saliency index also contains other index data greater than 0.5, it can also be output as a target motion object at the same time.
[0087] Step S291: determine a target image according to the target moving object.
[0088] After determining the target image, since the target image is a grayscale image with a length × width × 1, and the pan-focus image is an RGB image with a length × width × 3, the target image and the pan-focus image can be spliced to form an image with a length × width × 4. When using this image, the user can extract the data information therein according to their needs to form the desired image.
[0089] For example, Figure 3Provided is an implementation example diagram for determining a motion saliency index, wherein a light field camera 1 collects and outputs a light field image 11, an event camera 2 collects and outputs a grayscale image 21, a depth map 111 and a pan-focus map 112 are determined through the light field image 11; by inputting the pan-focus map 112 and the grayscale image 21 into a U-net neural network segmentation model 3, a current regional distribution image 31 output by the U-net neural network segmentation model 3 and containing at least one dynamic object and a motion speed 32 corresponding to each dynamic object are obtained; a depth distribution matrix 41 and a size distribution matrix 42 are determined through the current regional distribution image 31 and the depth map 111, a speed distribution matrix 43 is determined through the motion speed of each dynamic object, a new object matrix 44 is determined by comparing the current regional distribution image 31 with the previous regional distribution image 33 to determine whether the dynamic object is a new object, and a motion saliency index 51 is obtained and output by weighted summing the depth distribution matrix 41, the size distribution matrix 42, the speed distribution matrix 43 and the new object matrix 44 according to corresponding weights, and the pan-focus map 112 is output at the same time.
[0090] The embodiment of the present invention provides a light field imaging method for cross-scale adaptive mapping, which solves the problem of frequent and large-scale image acquisition caused by slight changes when acquiring images of a monitoring area. By analyzing the light field image and the grayscale image, each dynamic object is determined, and then the depth distribution matrix, the size distribution matrix, the speed distribution matrix and the newly added object matrix are determined. By adaptively setting different weights for different matrices, a motion significance index is obtained, and each dynamic object is screened according to different focus points to obtain a target moving object. The dynamic objects that are not needed or expected to be paid attention to in the image can be effectively filtered. When the target moving object is not included in the image, the image does not need to be processed, the data processing amount is effectively reduced, the difficulty of subsequent data processing is reduced, and the data analysis speed is improved. By identifying the target moving object for subsequent data processing, the problem of information loss caused by excessive data volume can be effectively avoided, and the adaptive separation of background information and the target moving object of interest is achieved, so as to achieve the goal of high-throughput, high-efficiency and highly targeted information acquisition and extraction. The algorithm is concise and easy to implement. The light field camera is controlled by the triggering number of the triggering event sent by the event camera, which effectively avoids the collection of static redundant information and reduces the overhead of subsequent data storage and analysis.
[0091] Embodiment 3
[0092] Figure 4 This is a schematic structural diagram of a light field imaging device for cross-scale adaptive mapping provided in Embodiment 3 of the present invention. The device includes: an acquisition module 61, an indicator determination module 62, an object screening module 63 and an image determination module 64.
[0093] Among them, the acquisition module 61 is used to acquire the light field image captured by the light field camera and the grayscale image captured by the event camera, and determine the current regional distribution image containing at least one dynamic object and the motion speed corresponding to each of the dynamic objects based on the light field image and the grayscale image; the indicator determination module 62 is used to determine the motion significance index based on the light field image, the current regional distribution image and the motion speed corresponding to each of the dynamic objects; the object screening module 63 is used to screen each of the dynamic objects according to the motion significance index to obtain the target motion object; the image determination module 64 is used to determine the target image based on the target motion object.
[0094] The embodiment of the present invention provides a light field imaging device for cross-scale adaptive mapping, which solves the problem of frequent and large-scale image acquisition and processing due to small changes when acquiring images of a monitoring area. By analyzing the image, each dynamic object in the image is determined, and then a motion significance index is determined. According to the motion significance index, each dynamic object is screened to obtain a target motion object, which can effectively filter out dynamic objects that are not needed or expected to be paid attention to in the image. When the image does not contain the target motion object, there is no need to process the image, which effectively reduces the data processing amount, reduces the difficulty of subsequent data processing, and improves the data analysis speed. By identifying the target motion object for subsequent data processing, the problem of information loss caused by excessive data volume can be effectively avoided. The target motion object can be intuitively observed through the target image, which is convenient for subsequent data processing or execution of other operations.
[0095] Furthermore, the device also includes:
[0096] An accumulation module, used for receiving the trigger event sent by the event camera and accumulating the triggering times of the trigger event and adding 1;
[0097] A monitoring module, configured to control the light field camera to collect a light field image when it is detected that the triggering number is greater than a preset event threshold;
[0098] The grayscale image determination module is used to determine the grayscale image according to the event information of each triggering event.
[0099] Furthermore, the acquisition module 61 includes:
[0100] A pan-focus image determining unit, configured to determine a pan-focus image based on the light field image in combination with a preset algorithm;
[0101] A speed determination unit is used to combine the pan-focus image and the grayscale image with a predetermined neural network model to determine a current regional distribution image containing at least one dynamic object and a motion speed corresponding to each of the dynamic objects.
[0102] Furthermore, the indicator determination module 62 includes:
[0103] a matrix determination unit, configured to determine at least one motion parameter matrix according to the light field image, the current regional distribution image and the motion speed corresponding to each of the dynamic objects;
[0104] The index determination unit is used to determine the sum of the products of each motion parameter matrix and the corresponding weight as the motion significance index.
[0105] Furthermore, when the motion parameter matrix is a speed distribution matrix, the matrix determination unit is specifically used to: determine the regional pixel point set where each of the dynamic objects is located based on the current regional distribution image, the regional pixel point set including at least one regional pixel point; determine the motion speed of each of the dynamic objects as the target speed of each regional pixel point in the corresponding regional pixel point set, and form a speed distribution matrix based on the positions of all regional pixel points and the corresponding target speeds.
[0106] Furthermore, when the motion parameter matrix is a depth distribution matrix, the matrix determination unit is specifically used to: determine a depth map based on the light field image in combination with a preset algorithm; determine the depth of each regional pixel point in each regional pixel point set based on the depth map, and determine the average value of each depth as the object depth of the corresponding dynamic object; and form a depth distribution matrix based on the positions of all regional pixel points and the corresponding object depths.
[0107] Furthermore, when the motion parameter matrix is a size distribution matrix, the matrix determination unit is specifically used to: determine the number of horizontal pixels corresponding to each of the dynamic objects on the image sensor of the light field camera; determine the actual size of each of the dynamic objects according to the light field camera parameters, the object depth and the number of horizontal pixels corresponding to each of the dynamic objects combined with a predetermined formula; and form a size distribution matrix according to the positions of all area pixel points and the corresponding actual sizes.
[0108] Furthermore, when the motion parameter matrix is a newly added object matrix, the matrix determination unit is specifically used to: determine the newly added object image according to the current regional distribution image and the previous regional distribution image; determine the newly added object matrix according to the position and pixel value of each pixel point in the newly added object image.
[0109] Furthermore, the motion significance index includes at least one index data, and the object screening module 63 is specifically used to: determine the dynamic object corresponding to each index data greater than a preset output threshold as the target motion object.
[0110] The light field imaging device for cross-scale adaptive mapping provided in the embodiment of the present invention can execute the light field imaging method for cross-scale adaptive mapping provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.
[0111] It is worth noting that in the above-mentioned embodiment of the light field imaging device for cross-scale adaptive mapping, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0112] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A light field imaging method for cross-scale adaptive mapping, characterized in that: include: Acquire a light field image captured by a light field camera and a grayscale image captured by an event camera, and determine a current regional distribution image containing at least one dynamic object and a motion speed corresponding to each of the dynamic objects according to the light field image and the grayscale image; Determining a motion saliency index according to the light field image, the current regional distribution image and the motion speed corresponding to each of the dynamic objects; Screening each of the dynamic objects according to the motion significance index to obtain a target motion object; Determine a target image according to the target moving object; The determining of the motion significance index according to the light field image, the current regional distribution image and the motion speed corresponding to each of the dynamic objects comprises: Determine at least one motion parameter matrix according to the light field image, the current regional distribution image and the motion speed corresponding to each of the dynamic objects; Determining the sum of the products of each of the motion parameter matrices and the corresponding weights as a motion significance index; When the motion parameter matrix is a size distribution matrix, correspondingly, determining the size distribution matrix includes: Determine the number of horizontal pixels corresponding to each of the dynamic objects on the image sensor of the light field camera; Determine the actual size of each of the dynamic objects according to the light field camera parameters, the object depth and the number of horizontal pixels corresponding to each of the dynamic objects in combination with a predetermined formula; A size distribution matrix is formed according to the positions of the pixels in all regions and the corresponding actual sizes.
2. The method according to claim 1, characterized in that Before acquiring the light field image acquired by the light field camera and the grayscale image acquired by the event camera, the method further includes: Receive the trigger event sent by the event camera and add 1 to the triggering number of the trigger event; When it is detected that the triggering number is greater than a preset event threshold, controlling the light field camera to collect a light field image; A grayscale image is determined according to the event information of each of the triggering events.
3. The method according to claim 1, characterized in that The determining, according to the light field image and the grayscale image, a current regional distribution image containing at least one dynamic object and a motion speed corresponding to each of the dynamic objects comprises: Determine a pan-focus image based on the light field image in combination with a preset algorithm; The pan-focus image and the grayscale image are combined with a predetermined neural network model to determine a current regional distribution image containing at least one dynamic object and a motion speed corresponding to each of the dynamic objects.
4. The method according to claim 1, characterized in that: When the motion parameter matrix is a velocity distribution matrix, correspondingly, determining the velocity distribution matrix includes: Determine a regional pixel point set where each of the dynamic objects is located according to the current regional distribution image, wherein the regional pixel point set includes at least one regional pixel point; The movement speed of each of the dynamic objects is determined as the target speed of each regional pixel point in the corresponding regional pixel point set, and a speed distribution matrix is formed according to the positions of all regional pixel points and the corresponding target speeds.
5. The method according to claim 4, characterized in that When the motion parameter matrix is a depth distribution matrix, correspondingly, determining the depth distribution matrix includes: Determine a depth map based on the light field image in combination with a preset algorithm; Determine the depth of each regional pixel point in each regional pixel point set according to the depth map, and determine the average value of each depth as the object depth of the corresponding dynamic object; A depth distribution matrix is formed according to the positions of all the pixel points in the area and the corresponding object depths.
6. The method according to claim 1, characterized in that When the motion parameter matrix is a newly added object matrix, correspondingly, determining the newly added object matrix includes: Determining a newly added object image according to the current area distribution image and the previous area distribution image; The newly added object matrix is determined according to the position and pixel value of each pixel point in the newly added object image.
7. The method according to claim 1, characterized in that The motion significance index includes at least one indicator data. Accordingly, the screening of each of the dynamic objects according to the motion significance index to obtain the target motion object includes: The dynamic objects corresponding to the indicator data greater than the preset output threshold are determined as the target moving objects.
8. A light field imaging device for cross-scale adaptive mapping, characterized in that: include: an acquisition module, configured to acquire a light field image acquired by a light field camera and a grayscale image acquired by an event camera, and determine a current regional distribution image containing at least one dynamic object and a motion speed corresponding to each of the dynamic objects according to the light field image and the grayscale image; An index determination module, used to determine a motion significance index according to the light field image, the current regional distribution image and the motion speed corresponding to each of the dynamic objects; An object screening module, used for screening each of the dynamic objects according to the motion significance index to obtain a target motion object; An image determination module, used to determine a target image according to the target moving object; Indicator determination module, including: a matrix determination unit, configured to determine at least one motion parameter matrix according to the light field image, the current regional distribution image and the motion speed corresponding to each of the dynamic objects; An index determination unit, used to determine the sum of the products of each motion parameter matrix and the corresponding weight as a motion significance index; When the motion parameter matrix is a size distribution matrix, it includes: a matrix determination unit, which is used to determine the number of horizontal pixels corresponding to each of the dynamic objects on the image sensor of the light field camera; determine the actual size of each of the dynamic objects according to the light field camera parameters, the object depth and the number of horizontal pixels corresponding to each of the dynamic objects combined with a predetermined formula; and form a size distribution matrix according to the positions of all area pixel points and the corresponding actual sizes.
