High-speed visual scanning system and method based on liquid lens

Abstract

The present invention provides a high-speed visual scanning system and method based on a liquid lens, belonging to the field of visual scanning technology. The system includes: a relationship determination module, which is used to obtain multiple images of a scene to be scanned at the same time based on a liquid lens, and determine the mapping relationship between the actual shooting range and depth data of each image; a scene fusion module, which is used to perform scene fusion on all images based on the mapping relationship of each image, and obtain a fused image and a scene fusion strategy; a sequence generation module, which is used to generate a high-frame image sequence of the scene to be scanned based on the fused image and the scene fusion strategy; and a visual scanning module, which is used to determine the high-speed visual scanning results of the scene to be scanned in the high-frame image sequence based on deep learning; and is used to achieve high-speed and accurate scanning of objects contained in the scene to be scanned.

Description

Technical Field

[0001] The present invention relates to the field of visual scanning technology, and in particular to a high-speed visual scanning system and method based on a liquid lens. Background Art

[0002] In recent years, emerging concepts such as driverless cars and smart cities have gradually become a reality. These advancements not only rely on the rapid development of network bandwidth, big data, and computing power, but also require the upgrading and transformation of various supporting hardware. Image acquisition equipment, the eyes of the intelligent age, plays a vital role in many fields, including urban transportation, public security monitoring, and industrial automation. The main development trends for image acquisition equipment are wider viewing angles, faster response times, and higher-quality imaging. Traditional mechanical zoom lenses achieve different focal lengths by mechanically shifting the distance between two lenses. Liquid zoom lenses, by contrast, offer advantages such as high-speed zoom, compact structure, fast response times, high reliability, and enhanced stability.

[0003] However, existing visual surveillance systems have many other defects, which means that existing surveillance cannot fully monitor every corner in real time. In order to further improve the level of surveillance, it is important to use the images captured by liquid lenses to achieve high-speed visual scanning of targets in the scene.

[0004] Therefore, the present invention proposes a high-speed visual scanning system and method based on liquid lens. Summary of the Invention

[0005] The present invention provides a high-speed visual scanning system and method based on a liquid lens, which is used to achieve scene fusion between multiple visual images based on the mapping relationship between the actual shooting range and depth data of the images obtained using the liquid lens, and further generate a high-frame number image sequence in time sequence, and then combine deep learning to achieve high-speed visual scanning of targets in the scene.

[0006] The present invention provides a high-speed visual scanning system based on a liquid lens, comprising:

[0007] A relationship determination module is used to obtain multiple images of the scene to be scanned at the same time based on the liquid lens, and determine the mapping relationship between the actual shooting range and depth data of each image;

[0008] The scene fusion module is used to perform scene fusion on all captured images based on the mapping relationship of each captured image to obtain a fused image and scene fusion strategy;

[0009] A sequence generation module is used to generate a high-frame image sequence of the scene to be scanned based on the fused image and scene fusion strategy;

[0010] The visual scanning module is used to determine the high-speed visual scanning results of the scene to be scanned in the high-frame image sequence based on deep learning.

[0011] Preferably, the relationship determination module includes:

[0012] An image capture submodule, configured to capture multiple images of the scene to be scanned at the same time based on a liquid lens with multiple preset shooting angles;

[0013] A 3D restoration submodule is used to determine the depth data and 2D outline of each captured image, and perform 3D restoration of the actual shooting range of the captured image based on the depth data and 2D outline to obtain a 3D coordinate set of the actual shooting range of the captured image;

[0014] The relationship determination submodule is used to regard the three-dimensional coordinate set as a mapping relationship between the actual shooting range of the shot image and the depth data.

[0015] Preferably, the scene fusion module includes:

[0016] A fusion preparation submodule is used to determine, based on the mapping relationship, a boundary combination to be fused between all images captured at the same moment, and to determine a partial mapping relationship of the boundaries to be fused in the boundary combination to be fused in the corresponding captured images;

[0017] The image correction submodule is used to perform high-detail fusion correction on the captured images belonging to the to-be-fused boundary in the to-be-fused boundary combination, and obtain multiple corrected images at the same moment and a correction strategy for each captured image;

[0018] The scene fusion submodule is used to fuse multiple corrected images at the same time to obtain a fused image and determine the relative position of each corrected image in the fused image;

[0019] The strategy generation submodule is used to obtain a scene fusion strategy based on the correction strategy of the correction image of the captured image obtained by each liquid lens and the relative position in the fusion image.

[0020] Preferably, the image correction submodule includes:

[0021] A region determining unit, configured to determine a partial image region of the boundary to be fused in the boundary combination to be fused in the corresponding captured image;

[0022] a region replacement unit, configured to evaluate the detail completeness of each partial image region, replace the partial image region of another to-be-fused boundary in the to-be-fused boundary combination in the corresponding captured image with the partial image region with the maximum detail completeness, obtain a replacement region, and use the partial image region with the maximum detail completeness as a replacement source for the replacement region;

[0023] A transition correction unit is used to perform image parameter transition correction on the adjacent image area of ​​the replacement area in the corresponding captured image based on the partial image area with the maximum detail completeness, so as to obtain a high-detail fusion correction result and a transition correction strategy of the captured image at the current boundary to be fused;

[0024] A correction summarization unit, configured to fuse the correction results at all boundaries to be fused based on the multiple images captured at the same moment, thereby obtaining multiple corrected images at the same moment;

[0025] The strategy determining unit is configured to use the replacement area and the replacement source of the captured image and the transition correction strategy of the image area adjacent to the replacement area as the correction strategy of the captured image.

[0026] Preferably, the sequence generation module includes:

[0027] A time determination submodule is used to determine a plurality of time moments that are continuous with the time moment of the fused image and treat them as a plurality of continuous time moments;

[0028] A scene fusion submodule is used to obtain multiple images of the scene to be scanned at each continuous moment, and perform scene fusion on the multiple images at the continuous moments based on the scene fusion strategy to obtain a fused image of the scene to be scanned at each continuous moment;

[0029] The interpolation processing submodule is used to perform interpolation processing on all fused images to obtain a high-frame image sequence.

[0030] Preferably, the frame insertion processing submodule includes:

[0031] A contour recognition unit, used for performing contour recognition on all fused images to obtain the contour contained in each fused image;

[0032] A contour matching unit, configured to match contours contained in different fused images based on the shapes of the contours to obtain contours that match each other;

[0033] a displacement determining unit, configured to determine a global pixel displacement of adjacent fused images based on coordinate representations of all mutually matching contours in the adjacent fused images in the corresponding fused images;

[0034] an interpolation processing unit, configured to perform interpolation processing on adjacent fused images based on global pixel displacement of adjacent fused images to obtain interpolated images;

[0035] The image sorting unit is used to sort all fused images and interpolated images to obtain a high-frame image sequence.

[0036] Preferably, the displacement determining unit comprises:

[0037] a sequence generation subunit, configured to determine the curvature of each contour point in the mutually matching contours based on the coordinate representations of all mutually matching contours in the adjacent fused images in the corresponding fused images, and to generate a local curvature sequence of the contour points by combining the curvatures of multiple contour points adjacent to and preceding the current contour point in the contour;

[0038] a contour point matching subunit, configured to perform contour point matching on the mutually matching contours based on the Euclidean distance between the local curvature sequences of two contour points respectively belonging to the mutually matching contours, and obtain a matching contour point combination;

[0039] a displacement determination subunit, configured to determine a pixel displacement of the contour point combination based on a coordinate value of each contour point in the contour point combination in the corresponding fused image;

[0040] The global inference subunit is used to infer the global pixel displacement of adjacent fused images based on the pixel displacement of all contour point combinations in all mutually matching contours in adjacent fused images.

[0041] Preferably, the method for the global inference subunit to infer the global pixel displacement of adjacent fused images based on the pixel displacement of all contour point combinations in all mutually matching contours in adjacent fused images includes:

[0042] The remaining pixel positions in the adjacent fused images, except for the contour points contained in all matching contour point combinations, are regarded as non-contour point positions;

[0043] Based on the pixel displacement of all contour point combinations in all matching contours in adjacent fused images, pixel displacement interpolation processing is performed on row pixels in adjacent fused images to obtain row interpolation pixel displacement of non-contour point positions;

[0044] Based on the pixel displacement of all contour point combinations in all matching contours in adjacent fused images, pixel displacement interpolation processing is performed on column pixels in adjacent fused images to obtain column interpolation pixel displacement of non-contour point positions;

[0045] Determine the distance between the non-contour point position and the source contour point in the row interpolation process, and use it as the row interpolation distance of the non-contour point position; and determine the distance between the non-contour point position and the source contour point in the column interpolation process, and use it as the column interpolation distance of the non-contour point position;

[0046] Based on the row interpolation pixel displacement and row interpolation distance of the non-contour point position and the column interpolation pixel displacement and column interpolation distance, the pixel displacement of each non-contour point position in the adjacent fused image is calculated;

[0047] The pixel displacement of all contour point combinations and the pixel displacement of all non-contour point positions in all matching contours in adjacent fused images are regarded as the global pixel displacement of adjacent fused images.

[0048] Preferably, the visual scanning module includes:

[0049] The recognition scanning submodule is used to perform visual scanning on each fused image in the high-frame image sequence, obtain clear scan objects and blurred scan objects, and identify the required scanning information of the clear scan objects;

[0050] A sequence inference submodule is used to determine the required scanning information of the blurred scan object in the high-frame image sequence based on deep learning;

[0051] The feature aggregation submodule is used to treat the required scanning information of all scanned objects as the high-speed visual scanning result of the scene to be scanned;

[0052] The scanning objects include clear scanning objects and fuzzy scanning objects.

[0053] The present invention provides a high-speed visual scanning method based on a liquid lens, which is applied to any one of the high-speed visual scanning systems based on a liquid lens in Examples 1 to 9, comprising:

[0054] S1: Acquire multiple images of the scene to be scanned at the same time based on the liquid lens, and determine the mapping relationship between the actual shooting range and depth data of each image;

[0055] S2: Based on the mapping relationship of each captured image, all captured images are subjected to scene fusion to obtain a fused image and scene fusion strategy;

[0056] S3: Generate a high-frame image sequence of the scene to be scanned based on the fused image and scene fusion strategy;

[0057] S4: Based on deep learning, high-speed visual scanning results of the scene to be scanned are determined in the high-frame image sequence.

[0058] The beneficial effects of the present invention compared to the prior art are: based on the mapping relationship between the actual shooting range and depth data of the captured image obtained by using a liquid lens, scene fusion between multiple visual captured images is achieved, and a high-frame-number image sequence in time sequence is further generated, and then combined with deep learning to achieve high-speed visual scanning of targets in the scene.

[0059] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0060] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0062] Figure 1 Schematic diagram of a high-speed visual scanning system based on a liquid lens in an embodiment of the present invention;

[0063] Figure 2 Schematic diagram of a relationship determination module of a liquid lens in an embodiment of the present invention;

[0064] Figure 3 This is a flow chart of a high-speed visual scanning method based on a liquid lens in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0066] Example 1:

[0067] The present invention provides a high-speed visual scanning system based on liquid lens, referring to Figure 1 ,include:

[0068] a relationship determination module for acquiring, based on a liquid lens, multiple images of a scene to be scanned (i.e., an actual scene requiring high-speed visual scanning using the liquid lens-based high-speed visual scanning system of this embodiment) at the same instant (each captured image is captured by the liquid lens at a preset viewing angle and posture, and includes a portion of the scene to be scanned), and determining a mapping relationship between an actual capturing range (i.e., the actual spatial range corresponding to the scene details contained in the captured image) and a depth (i.e., the depth value of each pixel in the captured image) of each captured image (i.e., the mapping relationship between the actual position point corresponding to each pixel in the captured image within the actual capturing range and the depth value of the pixel point, where the depth value is the actual distance between the actual position point and the captured position point);

[0069] The scene fusion module is used to perform scene fusion on all captured images based on the mapping relationship of each captured image, and obtain a fused image (i.e., an image obtained by fusing all captured images acquired at the same time according to the actual scene position) and a scene fusion strategy (i.e., a strategy for how to obtain a fused image by fusing captured images);

[0070] A sequence generation module is used to generate a high-frame image sequence of the scene to be scanned based on the fused image and scene fusion strategy (i.e., an image sequence of the fused image at each moment in a continuous period including the current moment);

[0071] The visual scanning module is used to determine the high-speed visual scanning results of the scene to be scanned in the high-frame image sequence based on deep learning (that is, based on deep learning, target recognition is performed on the high-frame image sequence, and the image features of the identified target in the high-frame image sequence are used as the high-speed visual scanning results. The image features can be, for example, the corresponding image area in the high-frame image sequence, or a sequence of image areas (including its movements), or the distance between the target and a preset position, etc.).

[0072] Based on the mapping relationship between the actual shooting range and depth data of the captured images obtained using a liquid lens, scene fusion between multiple visual images is achieved, and a high-frame-rate image sequence is further generated in time sequence. Combined with deep learning, high-speed visual scanning of targets in the scene is achieved.

[0073] Example 2:

[0074] On the basis of Example 1, the relationship determination module, referring to Figure 2 ,include:

[0075] An image capture submodule is used to obtain multiple images of the scene to be scanned at the same time based on liquid lenses with multiple preset shooting angles (each liquid lens corresponds to a preset shooting angle and an image captured at a moment. The preset shooting angles are manually set, and the field of view of all shooting angles can cover the entire scene to be scanned);

[0076] a three-dimensional restoration submodule for determining depth data (i.e., data containing the depth value of each pixel in the captured image, wherein the depth data may be determined by performing coordinate transformation on the captured image to obtain its point cloud data, and obtaining the depth data based on the point cloud data) and a two-dimensional contour (i.e., the contour contained in the captured image obtained after contour recognition on the captured image), performing three-dimensional restoration of the actual shooting range of the captured image based on the depth data and the two-dimensional contour (i.e., constructing a three-dimensional model of the scene contained in the captured image based on the depth data and the two-dimensional contour), and obtaining a three-dimensional coordinate set of the actual shooting range of the captured image (i.e., a set of coordinate values ​​of all position points in the three-dimensional model constructed in the three-dimensional restoration step);

[0077] The relationship determination submodule is used to regard the three-dimensional coordinate set as a mapping relationship between the actual shooting range of the shot image and the depth data.

[0078] Based on the depth data of the images captured at the same time obtained from multiple preset shooting angles and the two-dimensional contours contained therein, three-dimensional restoration of the scene details contained in the captured images is achieved. Based on the coordinate values ​​of the model in the three-dimensional restoration results, the mapping relationship between the actual position point corresponding to each pixel point in the captured image in the actual shooting range and the depth value of the pixel point is accurately represented.

[0079] Example 3:

[0080] Based on Example 1, the scene fusion module includes:

[0081] A fusion preparation submodule is used to determine, based on the mapping relationship, a combination of boundaries to be fused between all images captured at the same time (i.e., regions formed by pixels with the same mapping relationship between all images captured at the same time are considered to be boundaries to be fused, and a combination of regions that are boundaries to be fused is a combination of boundaries to be fused), and to determine a partial mapping relationship of the boundaries to be fused in the combination of boundaries to be fused in the captured images to which they belong (i.e., a mapping relationship between the actual position corresponding to the pixel point of the boundary to be fused in the captured image to which it belongs and the depth value of the pixel point in the actual shooting range);

[0082] The image correction submodule is used to perform high-detail fusion correction on the captured images belonging to the to-be-fused boundary in the to-be-fused boundary combination (i.e., the process of performing fusion correction based on the principle of preserving as much detail as possible), obtain multiple corrected images at the same time (i.e., images obtained after performing high-detail fusion correction on the captured images) and a correction strategy for each captured image (i.e., the strategy for how to perform high-detail fusion correction on the captured images to obtain the corresponding corrected image);

[0083] A scene fusion submodule is used to fuse multiple corrected images at the same time to obtain a fused image and determine the relative position of each corrected image in the fused image (for example, using the two-dimensional coordinate representation of each corrected image in the two-dimensional coordinate system corresponding to the fused image to represent the relative position);

[0084] The strategy generation submodule is used to obtain a scene fusion strategy based on the correction strategy of the correction image of the captured image obtained by each liquid lens and the relative position in the fusion image.

[0085] Based on the mapping relationship, the boundary combination to be fused between the images taken at the same moment is determined, and through high-detail fusion correction of the boundary combination to be fused, the correction of the captured images is achieved on the principle of retaining as many details as possible, providing preparatory conditions for the subsequent scene fusion between the images taken at the same moment, so as to achieve seamless scene fusion between multiple images taken at the same moment, and obtaining the scene fusion strategy based on the correction strategy in the correction process and the relative position of the captured images obtained by each liquid lens in the fused image during fusion, providing a basis for the subsequent scene fusion process at other continuous moments, thereby increasing the scene fusion efficiency at continuous moments.

[0086] Example 4:

[0087] Based on Example 3, the image correction submodule includes:

[0088] A region determining unit, configured to determine a partial image region of the boundary to be fused in the boundary combination to be fused in the corresponding captured image;

[0089] a region replacement unit, configured to evaluate the detail completeness of each partial image region (the detail completeness may be obtained by evaluating using a pre-trained detail completeness evaluation model, or by determining the detail completeness using manually evaluated detail completeness input, wherein the detail completeness is a numerical value representing the degree of completeness of image details in the partial image region), replace the partial image region in the captured image corresponding to another boundary to be fused in the boundary combination to be fused with the partial image region with the maximum detail completeness (i.e., replace the partial image region A in the captured image corresponding to another boundary to be fused in the boundary combination to be fused with the partial image region B with the maximum detail completeness), obtain a replacement region (i.e., the replaced partial image region B), and use the partial image region with the maximum detail completeness as a replacement source for the replacement region (i.e., the partial image region A that replaces the partial image region A);

[0090] a transition correction unit for performing image parameter transition correction on an adjacent image region C of the replacement region in the corresponding captured image based on a partial image region with maximum detail completeness (i.e., adjusting image parameters of the adjacent image region C so that the transition between them and the image parameters of the replacement region is smoother, such as contrast and brightness), obtaining a high-detail fusion correction result of the captured image at the current boundary to be fused (which is the result obtained after performing image parameter transition correction on the adjacent image region of the captured image at the current boundary to be fused) and a transition correction strategy (i.e., a strategy for correcting the image parameters of each pixel in the adjacent image region to the image parameter of the corresponding pixel in the image in the high-detail fusion correction result);

[0091] A correction summarization unit, configured to fuse the correction results at all boundaries to be fused based on the multiple images captured at the same moment, thereby obtaining multiple corrected images at the same moment;

[0092] The strategy determining unit is configured to use the replacement area and the replacement source of the captured image and the transition correction strategy of the image area adjacent to the replacement area as the correction strategy of the captured image.

[0093] By evaluating the completeness of details in the partial image area corresponding to the boundary to be fused in the captured image, the partial image area of ​​the side with higher detail completeness is used to replace the partial image area of ​​the other side in the boundary combination to be fused, and the image parameters of the adjacent image areas of the replaced partial image area are adjusted to make the connection between them and the replaced area higher. In this way, more image details are retained as much as possible in the correction link before scene fusion of the captured images, and the degree of image fragmentation in the corrected image is greatly reduced through high-detail fusion correction.

[0094] Example 5:

[0095] Based on Example 1, the sequence generation module includes:

[0096] A time determination submodule is used to determine multiple time moments that are continuous with the time moment of the fused image and treat them as multiple continuous time moments (i.e., multiple time moments t-1, t-2, t-3, ..., that are before the time moment t of the fused image and are sequentially continuous with the time moment of the fused image);

[0097] A scene fusion submodule is used to obtain multiple images of the scene to be scanned at each consecutive moment, and perform scene fusion on the multiple images at the consecutive moments based on the scene fusion strategy (i.e., the process of performing high-detail fusion correction on the images according to the operation method included in the scene fusion strategy), thereby obtaining a fused image of the scene to be scanned at each consecutive moment;

[0098] The interpolation processing submodule is used to perform interpolation processing on all fused images to obtain a high-frame image sequence.

[0099] Based on the scene fusion strategy used to generate the fused image at the current moment, efficient and accurate fusion of all captured images at other consecutive moments is achieved. In addition, all fused images that change over time are interpolated to increase the total number of frames of all fused images, providing rich image resources for subsequent scanning of objects contained in the scene to be scanned.

[0100] Example 6:

[0101] Based on Example 5, the frame insertion processing submodule includes:

[0102] A contour recognition unit, configured to perform contour recognition on all fused images (based on a contour recognition algorithm such as a Canny detection operator) to obtain contours contained in each fused image;

[0103] a contour matching unit for matching contours contained in different fused images based on the shapes of the contours to obtain mutually matching contours (the matching method includes, for example, taking a pixel point in the contour that intersects the positive direction of the ordinate axis of the two-dimensional coordinate system as a starting pixel point, and determining a pixel point sequence obtained by sorting all pixel points in a single contour in a clockwise direction, sorting the curvatures of the contour at all pixel points to obtain a curvature sequence of the contour, calculating the similarity between the curvature sequences of two contours contained in different fused images, and treating two contours corresponding to two curvature sequences whose similarity exceeds a preset similarity threshold as mutually matching contours, wherein the method for calculating the similarity between two curvature sequences includes, for example, the following: a first step: calculating the ratio between the difference and the mean of the values ​​of all the same sorted ordinal numbers in the curvature sequence, treating the mean of the ratios corresponding to all the sorted ordinal numbers in the curvature sequence as the deviation between the two curvature sequences, and treating the difference between 1 and the deviation as the similarity between the two curvature sequences);

[0104] a displacement determining unit, configured to determine a global pixel displacement of adjacent fused images (i.e., a displacement of all pixel positions in adjacent fused images) based on coordinate representations of all mutually matching contours in the adjacent fused images in the corresponding fused images;

[0105] an interpolation processing unit, configured to perform interpolation processing on adjacent fused images based on global pixel displacement of the adjacent fused images to obtain an interpolation image (i.e., an image formed between adjacent fused images during the interpolation processing of the adjacent fused images);

[0106] The image sorting unit is used to sort all fused images and interpolated images to obtain a high-frame image sequence.

[0107] By performing contour recognition on the fused image and achieving contour matching in different fused images based on the formation of the recognized contours, the global pixel displacement of adjacent fused images is inferred based on the matching contours in adjacent fused images, and the adjacent fused images are interpolated based on the global pixel displacement to obtain a high-frame-number image sequence of the scene to be scanned.

[0108] Example 7:

[0109] Based on Example 6, the displacement determining unit includes:

[0110] a sequence generation subunit for determining the curvature of each contour point in the mutually matching contours based on the coordinate representations of all mutually matching contours in the adjacent fused images in the corresponding fused images, and generating a local curvature sequence of the contour point (e.g., a curvature sequence including the curvature of the contour point and the curvatures of the three contour points that are sequentially adjacent to the front of the contour point and the curvatures of the three contour points that are sequentially adjacent to the back of the contour point) by combining the curvatures of multiple contour points that are adjacent to the front of the contour point and the curvatures of the three contour points that are sequentially adjacent to the back of the contour point) in the contour;

[0111] a contour point matching subunit for performing contour point matching on the mutually matching contours (taking the contour point corresponding to the maximum Euclidean distance between each contour point in the mutually matching contours as the contour point matching the current contour point) based on the Euclidean distance between the local curvature sequences of two contour points respectively belonging to the mutually matching contours (the Euclidean distance between two local curvature sequences is calculated as follows: the square root of the sum of the squares of the differences between the values ​​of all identical ordinal numbers in the local curvature sequences is the Euclidean distance between the two), and obtaining a matching contour point combination (i.e., a combination consisting of the mutually matching contour points respectively belonging to the mutually matching contours);

[0112] a displacement determination subunit for determining a pixel displacement of the contour point combination based on the coordinate value of each contour point in the contour point combination in the corresponding fused image (i.e., taking the displacement from the coordinate value of the contour point in the contour point combination corresponding to the earlier time to the coordinate value of the contour point in the contour point combination corresponding to the later time as the pixel displacement);

[0113] The global inference subunit is used to infer the global pixel displacement of adjacent fused images based on the pixel displacement of all contour point combinations in all mutually matching contours in adjacent fused images.

[0114] Based on the curvature of continuous contour points in the local contour of the fused image, a local curvature sequence of the fused image is generated, and the Euclidean distance between the local curvature sequences is used as the basis to achieve accurate matching of contour points in mutually matching contours based on local contour shapes, and based on the matching results of contour points in mutually matching contours, the pixel displacement between mutually matching contour points in mutually matching contours is determined, and the global pixel displacement between adjacent fused images is inferred based on the pixel displacement of contour points.

[0115] Example 8:

[0116] Based on Example 7, the method for the global inference subunit to infer the global pixel displacement of adjacent fused images based on the pixel displacement of all contour point combinations in all mutually matching contours in adjacent fused images includes:

[0117] The remaining pixel positions in the adjacent fused images, except for the contour points contained in all matching contour point combinations, are regarded as non-contour point positions;

[0118] Based on the pixel displacements of all contour point combinations in all matching contours in adjacent fused images, pixel displacement interpolation processing is performed on row pixels (i.e., each row of pixels) in the adjacent fused images (i.e., using the pixel displacements of the pixel positions of the contour point combinations contained in each existing row in the adjacent fused images, interpolation processing is performed on the remaining pixel positions of the corresponding row in the adjacent fused images to determine the pixel displacements of the remaining pixel positions in the adjacent fused images), and row interpolation pixel displacements of non-contour point positions are obtained (i.e., the interpolated pixel displacements of the non-contour point positions are obtained by interpolating each row of pixels using the pixel displacements of the existing contour points);

[0119] Based on the pixel displacements of all contour point combinations in all matching contours in adjacent fused images, pixel displacement interpolation processing is performed on column pixels (i.e., each column of pixels) in the adjacent fused images (i.e., using the pixel displacements of the pixel positions of the contour point combinations contained in each existing column in the adjacent fused images, interpolation processing is performed on the remaining pixel positions of the corresponding column in the adjacent fused images to determine the pixel displacements of the remaining pixel positions in the adjacent fused images), and column interpolation pixel displacements of non-contour point positions are obtained (i.e., the interpolated pixel displacements of the non-contour point positions are obtained by interpolating each column of pixels using the pixel displacements of the existing contour points);

[0120] Determine the distance between the non-contour point position and the source contour point in the row interpolation process (i.e., for example, the pixel displacement of the non-contour point c after interpolation is determined by using the pixel displacement of the contour point a and the contour point b, and the contour point a and the contour point b are the source contour points in the interpolation process), and use it as the row interpolation distance of the non-contour point c (i.e., the distance between the non-contour point c and the contour point a and the contour point b respectively is the row interpolation distance). Also determine the distance between the non-contour point position and the source contour point in the column interpolation process, and use it as the column interpolation distance of the non-contour point position.

[0121] Based on the row interpolation pixel displacement and row interpolation distance, as well as the column interpolation pixel displacement and column interpolation distance, the pixel displacement of each non-contour point position in the adjacent fused image is calculated (the ratio of the row interpolation distance to the sum of the row interpolation distance and the column interpolation distance is determined, the difference between 1 and the ratio is used as the row interpolation weight, the ratio of the column interpolation distance to the sum of the row interpolation distance and the column interpolation distance is determined, the difference between 1 and the ratio is used as the column interpolation weight, and the average of the product of the row interpolation pixel displacement and the row interpolation weight and the product of the column difference displacement and the column difference weight is used as the pixel displacement of the non-contour point);

[0122] The pixel displacement of all contour point combinations and the pixel displacement of all non-contour point positions in all matching contours in adjacent fused images are regarded as the global pixel displacement of adjacent fused images.

[0123] By interpolating the pixel values ​​of non-contour points according to the row pixels and column pixels in the adjacent fused images, two interpolation results are obtained for each non-contour point. Then, based on the distance between the source contour point and the difference position in each interpolation process, the two interpolation results are weightedly summed according to the interpolation weight that decreases as the interpolation position increases, and the pixel displacement of each non-contour point is obtained, thus achieving a more accurate global estimation of the global pixel displacement of the adjacent fused images.

[0124] Example 9:

[0125] Based on Example 1, the visual scanning module includes:

[0126] The recognition and scanning submodule is used to perform a visual scan on each fused image in the high-frame image sequence to obtain clear scan objects (i.e., scan objects whose categories and required scanning information can be clearly identified, where the category is identified to determine whether it is an object category that needs to be identified, such as a vehicle or a person, and the required scanning information is, for example, the straight-line distance between the target and the shooting location, or whether the vehicle has violated a red light, etc.) and fuzzy scan objects (i.e., objects whose categories and / or required information cannot be clearly identified), and to identify the required scanning information for the clear scan objects (the required scanning information is the information required for this high-speed visual scan, such as the straight-line distance between the target and the shooting location, or whether the vehicle has violated a red light, etc.);

[0127] A sequence inference submodule is used to determine the required scanning information of the blurred scan object in the high-frame image sequence based on deep learning;

[0128] The feature aggregation submodule is used to treat the required scanning information of all scanned objects as the high-speed visual scanning result of the scene to be scanned;

[0129] The scanning objects include clear scanning objects and fuzzy scanning objects.

[0130] In this embodiment, a visual scan is performed in the fused image using an object scanning model that is pre-trained based on a deep learning algorithm and a large number of image area samples that mark clear scanning objects and blurred scanning objects to obtain clear scanning objects and blurred scanning objects in the fused image. Then, based on a clear scanning object recognition model that is pre-trained based on a deep learning algorithm and a large number of image area samples that mark object categories and required scanning information, the image area corresponding to the clear scanning object is scanned to obtain the object category and required scanning information of the clear scanning object. Then, using a fuzzy scanning object recognition model that is pre-trained based on a deep learning algorithm and a large number of image area samples that mark the required scanning information of fuzzy scanning objects of different object types, the area where all fuzzy scanning objects are located is input into the fuzzy scanning object recognition model to determine the required scanning information of the fuzzy scanning objects.

[0131] By performing secondary recognition on the clear scan objects and blurred scan objects scanned using deep learning using different deep learning models, the object categories and required scanning information of all clear scan objects and blurred scan objects can be identified, thereby efficiently and accurately obtaining high-speed visual scanning results for all required scanning objects in the scene to be scanned.

[0132] Example 10:

[0133] The present invention provides a high-speed visual scanning method based on a liquid lens, which is applied to any one of the high-speed visual scanning systems based on a liquid lens in Examples 1 to 9, with reference to Figure 3 ,include:

[0134] S1: Acquire multiple images of the scene to be scanned at the same time based on the liquid lens, and determine the mapping relationship between the actual shooting range and depth data of each image;

[0135] S2: Based on the mapping relationship of each captured image, all captured images are subjected to scene fusion to obtain a fused image and scene fusion strategy;

[0136] S3: Generate a high-frame image sequence of the scene to be scanned based on the fused image and scene fusion strategy;

[0137] S4: Based on deep learning, high-speed visual scanning results of the scene to be scanned are determined in the high-frame image sequence.

[0138] Based on the mapping relationship between the actual shooting range and depth data of the captured images obtained using a liquid lens, scene fusion between multiple visual images is achieved, and a high-frame-rate image sequence is further generated in time sequence. Combined with deep learning, high-speed visual scanning of targets in the scene is achieved.

[0139] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. High-speed visual scanning system based on liquid lens, characterized by: include: A relationship determination module is used to obtain multiple images of the scene to be scanned at the same time based on the liquid lens, and determine the mapping relationship between the actual shooting range and depth data of each image; The scene fusion module is used to perform scene fusion on all captured images based on the mapping relationship of each captured image, and obtain the fused image and scene fusion strategy, including: A fusion preparation submodule is used to determine, based on the mapping relationship, a boundary combination to be fused between all images captured at the same moment, and to determine a partial mapping relationship of the boundaries to be fused in the boundary combination to be fused in the corresponding captured images; The image correction submodule is used to perform high-detail fusion correction on the captured images belonging to the to-be-fused boundary in the to-be-fused boundary combination, and obtain multiple corrected images at the same moment and a correction strategy for each captured image; The scene fusion submodule is used to fuse multiple corrected images at the same time to obtain a fused image and determine the relative position of each corrected image in the fused image; a strategy generation submodule for obtaining a scene fusion strategy based on a correction strategy of a correction image of a captured image acquired by each liquid lens and a relative position in the fused image; The sequence generation module is used to generate a high-frame image sequence of the scene to be scanned based on the fused image and scene fusion strategy, including: A time determination submodule is used to determine a plurality of time moments that are continuous with the time moment of the fused image and treat them as a plurality of continuous time moments; A scene fusion submodule is used to obtain multiple images of the scene to be scanned at each continuous moment, and perform scene fusion on the multiple images at the continuous moments based on the scene fusion strategy to obtain a fused image of the scene to be scanned at each continuous moment; The interpolation processing submodule is used to perform interpolation processing on all fused images to obtain a high-frame image sequence; The visual scanning module is used to determine the high-speed visual scanning results of the scene to be scanned in the high-frame image sequence based on deep learning.

2. The high-speed visual scanning system based on liquid lens according to claim 1, characterized in that: Relationship determination module, including: An image capture submodule, configured to capture multiple images of the scene to be scanned at the same time based on a liquid lens with multiple preset shooting angles; A 3D restoration submodule is used to determine the depth data and 2D outline of each captured image, and perform 3D restoration of the actual shooting range of the captured image based on the depth data and 2D outline to obtain a 3D coordinate set of the actual shooting range of the captured image; The relationship determination submodule is used to regard the three-dimensional coordinate set as a mapping relationship between the actual shooting range of the shot image and the depth data.

3. The high-speed visual scanning system based on liquid lens according to claim 1, characterized in that: Image correction submodule, including: A region determining unit, configured to determine a partial image region of the boundary to be fused in the boundary combination to be fused in the corresponding captured image; a region replacement unit, configured to evaluate the detail completeness of each partial image region, replace the partial image region of another to-be-fused boundary in the to-be-fused boundary combination in the corresponding captured image with the partial image region with the maximum detail completeness, obtain a replacement region, and use the partial image region with the maximum detail completeness as a replacement source for the replacement region; A transition correction unit is used to perform image parameter transition correction on the adjacent image area of ​​the replacement area in the corresponding captured image based on the partial image area with the maximum detail completeness, so as to obtain a high-detail fusion correction result and a transition correction strategy of the captured image at the current boundary to be fused; A correction summarization unit, configured to fuse the correction results at all boundaries to be fused based on the multiple images captured at the same moment, thereby obtaining multiple corrected images at the same moment; The strategy determining unit is configured to use the replacement area and the replacement source of the captured image and the transition correction strategy of the image area adjacent to the replacement area as the correction strategy of the captured image.

4. The high-speed visual scanning system based on liquid lens according to claim 1, characterized in that: The interpolation processing submodule includes: A contour recognition unit, used for performing contour recognition on all fused images to obtain the contour contained in each fused image; A contour matching unit, configured to match contours contained in different fused images based on the shapes of the contours to obtain contours that match each other; a displacement determining unit, configured to determine a global pixel displacement of adjacent fused images based on coordinate representations of all mutually matching contours in the adjacent fused images in the corresponding fused images; an interpolation processing unit, configured to perform interpolation processing on adjacent fused images based on global pixel displacement of adjacent fused images to obtain interpolated images; The image sorting unit is used to sort all fused images and interpolated images to obtain a high-frame image sequence.

5. The high-speed visual scanning system based on liquid lens according to claim 4, characterized in that: A displacement determination unit, comprising: a sequence generation subunit, configured to determine the curvature of each contour point in the mutually matching contours based on the coordinate representations of all mutually matching contours in the adjacent fused images in the corresponding fused images, and to generate a local curvature sequence of the contour points by combining the curvatures of multiple contour points adjacent to and preceding the current contour point in the contour; a contour point matching subunit, configured to perform contour point matching on the mutually matching contours based on the Euclidean distance between the local curvature sequences of two contour points respectively belonging to the mutually matching contours, and obtain a matching contour point combination; a displacement determination subunit, configured to determine a pixel displacement of the contour point combination based on a coordinate value of each contour point in the contour point combination in the corresponding fused image; The global inference subunit is used to infer the global pixel displacement of adjacent fused images based on the pixel displacement of all contour point combinations in all mutually matching contours in adjacent fused images.

6. The high-speed visual scanning system based on liquid lens according to claim 5, characterized in that: The method for the global inference subunit to infer the global pixel displacement of adjacent fused images based on the pixel displacement of all contour point combinations in all mutually matching contours in adjacent fused images includes: The remaining pixel positions in the adjacent fused images, except for the contour points contained in all matching contour point combinations, are regarded as non-contour point positions; Based on the pixel displacement of all contour point combinations in all matching contours in adjacent fused images, pixel displacement interpolation processing is performed on row pixels in adjacent fused images to obtain row interpolation pixel displacement of non-contour point positions; Based on the pixel displacement of all contour point combinations in all matching contours in adjacent fused images, pixel displacement interpolation processing is performed on column pixels in adjacent fused images to obtain column interpolation pixel displacement of non-contour point positions; Determine the distance between the non-contour point position and the source contour point in the row interpolation process, and use it as the row interpolation distance of the non-contour point position; and determine the distance between the non-contour point position and the source contour point in the column interpolation process, and use it as the column interpolation distance of the non-contour point position; Based on the row interpolation pixel displacement and row interpolation distance of the non-contour point position and the column interpolation pixel displacement and column interpolation distance, the pixel displacement of each non-contour point position in the adjacent fused image is calculated; The pixel displacement of all contour point combinations and the pixel displacement of all non-contour point positions in all matching contours in adjacent fused images are regarded as the global pixel displacement of adjacent fused images.

7. The high-speed visual scanning system based on liquid lens according to claim 1, characterized in that: Visual scanning module, including: The recognition scanning submodule is used to perform visual scanning on each fused image in the high-frame image sequence, obtain clear scan objects and blurred scan objects, and identify the required scanning information of the clear scan objects; A sequence inference submodule is used to determine the required scanning information of the blurred scan object in the high-frame image sequence based on deep learning; The feature aggregation submodule is used to treat the required scanning information of all scanned objects as the high-speed visual scanning result of the scene to be scanned; The scanning objects include clear scanning objects and fuzzy scanning objects.

8. A high-speed visual scanning method based on a liquid lens, characterized in that: The high-speed visual scanning system based on a liquid lens as claimed in any one of claims 1 to 7 comprises: S1: Acquire multiple images of the scene to be scanned at the same time based on the liquid lens, and determine the mapping relationship between the actual shooting range and depth data of each image; S2: Based on the mapping relationship of each captured image, all captured images are subjected to scene fusion to obtain a fused image and scene fusion strategy; S3: Generate a high-frame image sequence of the scene to be scanned based on the fused image and scene fusion strategy; S4: Based on deep learning, high-speed visual scanning results of the scene to be scanned are determined in the high-frame image sequence.

Images