Methods, Apparatus and Computer Readable Medium for Imaging - Patent application

Abstract

A method, apparatus, and computer readable medium for imaging are proposed, which includes the steps of: capturing and quantifying a first image of a target field of view at a first resolution and having a first bit width; capturing and differentially processing a second image of the target field of view at a second resolution and having a second bit width, the differential processing including, for a pixel point captured at the second resolution, quantifying a difference between the pixel point and an adjacent or nearby pixel point of the pixel point and obtaining the quantified difference as a value of the corresponding pixel point in the second image; and fusing the first and second images to obtain a third image, where the first resolution is lower than the second resolution and the first bit width is greater than the second bit width.

Description

[Technical field]

[0001] The present invention relates to a method, apparatus and computer readable medium for imaging, and more particularly to a method, apparatus and computer readable medium for ultra-high resolution real-time imaging. [Background technology]

[0002] Existing imaging means that can achieve super-high pixel counts (usually referring to more than 100 million pixels) include: (1) Super-resolution technology that increases the resolution of an image by enlarging a low-resolution image to a certain multiple using a conventional interpolation algorithm or an image restoration algorithm based on a neural network, but due to information loss during collection, the restored image is too smooth or contains false textures (especially when the enlargement multiple is too high), resulting in poor final imaging effect; (2) Multiple low-resolution partial images are captured and then used to stitch together a complete image with super-high pixel counts; such systems usually have characteristics such as large size, heavy weight, and complex circuit design; (3) Single-chip super-high pixel imaging, such systems can capture super-high pixel images with a single sensor chip, but they suffer from problems such as slow reading speed and large data volume, and cannot collect and transmit video in real time. All of the above methods directly collect information in the image domain, and the large amount of data involved makes it time-consuming to collect and transmit, so real-time video capture cannot be achieved. Summary of the Invention [Problem to be solved by the invention]

[0003] An object of the present invention is to provide a method, an apparatus, and a computer-readable medium that are capable of capturing images with ultra-high pixel counts quickly and even in real time. [Means for solving the problem]

[0004] In order to overcome the technical problems existing in the above technical background and develop a photographing and processing technique that can be used for super-high-resolution images, according to one aspect of the present invention, a novel image acquisition and fusion technique is proposed, which acquires two kinds of images with different resolutions and different bit widths for a target field of view (e.g., a high-resolution low-bit width difference image and a low-resolution high-bit width image), fuses the region of interest in the two kinds of images using an image fusion algorithm, and updates the complete image of the target field of view, thereby quickly and efficiently acquiring a clear super-high-resolution complete image of the target field of view.

[0005] According to one embodiment, the sparseness of the difference image (compared to within a frame) is utilized to compress and encode the collected high-resolution, low-bit-width difference image, thereby significantly reducing the amount of image data, improving the reading speed, reducing the transmission bandwidth, and contributing to the real-time collection of ultra-high-resolution images.

[0006] According to one embodiment, an image fusion algorithm based on a convolutional neural network can be used to fuse and reconstruct the high-resolution, low-bitwidth difference image and the low-resolution, high-bitwidth image.

[0007] According to one embodiment, an image recognition method may be used on the collected image, for example to determine and identify regions of interest contained therein based on morphological and / or dynamic features, and selectively output the morphological and dynamic features of the regions of interest.

[0008] According to one aspect of the present invention, the present invention proposes a method for imaging comprising the steps of: photographing a target field of view at a first resolution, quantifying and obtaining a first image having a first bit width; photographing the target field of view at a second resolution, quantifying and obtaining a second image having a second bit width, the differential processing including, for a pixel point photographed at the second resolution, quantifying a difference between the pixel point and an adjacent or nearby pixel point of the pixel point, and obtaining the quantified difference as a value of the corresponding pixel point in the second image; and fusing the first and second images to obtain a third image, wherein the first resolution is lower than the second resolution and the first bit width is greater than the second bit width.

[0009] According to one embodiment, the step of fusing the first image and the second image to obtain a third image further includes the steps of determining a region of interest of the target field of view in the second image using an image recognition method, obtaining a corresponding region in the first image that corresponds to the region of interest, and fusing the corresponding region of the first image with the region of interest in the second image to obtain the third image.

[0010] According to one embodiment, for a pixel point photographed at a second resolution, quantifying the differences between said pixel point and adjacent or nearby pixel points of said pixel point comprises quantifying said differences into a selected array.

[0011] According to one embodiment, the sequence is {-1, 0, +1} In this way, image data can be stored, processed and transmitted with a very small number of bits, i.e. with the most economical data resources.

[0012] According to an embodiment, the pixel point values ​​can be divided into five levels, for example, using two thresholds to compare pixel point values ​​in both positive and negative directions. For example, the array can be {-2,-1,0,+1,+2}. Such a five-value array approach is more useful for identification and fusion than a three-value array approach, and is especially useful for real-time and accurate identification and location of regions of interest.

[0013] According to one embodiment, the pixel point value can be divided into four levels, for example, using two thresholds to compare the pixel point value in both positive and negative directions. For example, the array can be {-2,-1,0,+1} or {-1,0,+1,+2}. Using this four-value array, only two bits are used to represent the pixel point value.

[0014] According to an embodiment of the present invention, the target field of view corresponding to the first image is the same as the target field of view corresponding to the second image. In practice, there is usually a time and / or space difference between the two. This is because there is a slight difference in the place and time when the two images were taken, so the two target fields of view do not necessarily overlap completely. However, as long as the overlapping areas of the two are the actual main parts of the images, or as long as both contain the area of ​​interest, they can be considered to be the same or equivalent fields of view. That is, the fields of view captured within a permissible range of time and / or space differences can be considered to be the same fields of view.

[0015] According to one embodiment, the method further comprises the steps of encoding and transmitting a second image prior to fusing, and decoding the second image after said transmission and using it for fusing.

[0016] According to one embodiment, the second image is encoded using a run-length encoding scheme, in which the bit sequence of the second image is encoded into a count sequence L, which records the number of times the repeating data is repeated, and a data sequence D, which records the repeating data itself.

[0017] According to one embodiment, the count sequence L is recorded using a Huffman code and the data sequence D is recorded using a fixed length code.

[0018] According to one embodiment, fusing the first image and the second image includes fusing the first image and the second image using a convolutional neural network.

[0019] According to one embodiment, the step of determining the area of ​​interest of the target field of view in the second image using an image recognition method includes a step of determining the area of ​​interest by combining images taken of the target field of view within a certain time range or previously stored images taken of the target field of view.

[0020] According to one embodiment, the method further includes a step of training the image recognition method using deep learning, where the image recognition method is trained using deep learning based on a selected target, spatial conditions when shooting, and manual labeling.

[0021] According to one embodiment, the step of fusing the region of interest and the corresponding region to obtain a third image further includes a step of fusing the region of interest and the corresponding region to obtain a third image only if the region of interest includes a specific object.

[0022] According to one embodiment, the method further comprises the step of outputting a third image after fusing, or the step of updating the complete image with the third image and then outputting an updated complete image.

[0023] According to one embodiment, a camera or video camera may be manufactured that implements the method of the present invention.

[0024] When the identification and reporting of the region of interest is limited by the capabilities of the system, especially the hardware, it may be difficult to report the image in real time. As a useful improvement, the camera or video camera of the present invention can extract the region of interest and report it if it is found to contain a specific object, in order to better meet the real-time requirement.

[0025] According to another aspect of the invention, the invention further proposes an apparatus for imaging, comprising an image capture component configured to capture and quantify a target field of view at a first resolution to obtain an image having a first bit width, and to capture and differentially process the target field of view at a second resolution to obtain a second image having a second bit width, the differential processing including, for a pixel point captured at the second resolution, quantifying a difference between the pixel point and an adjacent or nearby pixel point of the pixel point and obtaining the quantified difference as a value of the corresponding pixel point in the second image, and a data processing component coupled to the image capture component and configured to fuse the first image and the second image to obtain a third image, wherein the first resolution is lower than the second resolution and the first bit width is greater than the second bit width.

[0026] According to one embodiment, fusing the first image and the second image to obtain a third image further includes determining a region of interest of the target field of view in the second image using an image recognition method, obtaining a corresponding region of the first image that corresponds to the region of interest, and fusing the corresponding region of the first image with the region of interest in the second image to obtain a third image.

[0027] According to one embodiment, for a pixel point photographed at a second resolution, quantifying a difference between the pixel point and an adjacent or nearby pixel point of the pixel point includes quantifying the differences into a selected array.

[0028] According to one embodiment, the sequence is {-1,0,+1} or {-2,-1,0,+1,+2} or {-2,-1,0,+1} or {-1,0,+1,+2}.

[0029] According to one embodiment, the apparatus further includes an encoding component and a transmission component, the encoding component being coupled to the image capture component and configured to encode the second image before fusing, the transmission component being coupled to the encoding component and the data processing component and configured to transmit the encoded second image to the data processing component, the data processing component being further configured to decode the encoded second image after receiving the encoded second image for use in fusing.

[0030] According to one embodiment, the encoding component is further configured to encode the second image using a run-length encoding scheme, encoding a bit sequence of the second image into a count sequence L recording the number of repetitions of the repeating data and a data sequence D recording the repeating data itself.

[0031] According to one embodiment, the encoding component is further configured to record the counting sequence L using a Huffman code and to record the data sequence D using a fixed length code.

[0032] According to one embodiment, fusing the first image and the second image includes fusing the first image and the second image using a convolutional neural network.

[0033] According to one embodiment, the step of determining the area of ​​interest of the target field of view in the second image using an image recognition method includes determining the area of ​​interest by combining images taken of the target field of view within a certain time range or previously stored images taken of the target field of view.

[0034] According to one embodiment, the data processing component is further configured to train the image recognition method using deep learning, where deep learning is used to train the image recognition method based on selected targets, spatial conditions when photographing, and manual labeling.

[0035] According to another aspect of the invention, it is further proposed a non-transitory computer readable medium having recorded thereon a program code which, when executed by a computer, performs the method described above.

[0036] The technical solution of the present invention can reduce the number of bits of image collection to shorten the reading time, which helps to realize the real-time reading of ultra-high pixel difference images, and can effectively improve the accuracy and discrimination of images by implementing convolutional neural network operations in image fusion, so as to obtain clear ultra-high pixel images.In addition, the optimized on-chip compression encoding and decoding means of the present invention can compress image data in a targeted manner, which can greatly reduce the transmission bandwidth and realize real-time and high-speed image transmission. [Brief description of the drawings]

[0037] [Figure 1A] 2 is a schematic diagram illustrating a method flow for imaging according to an embodiment of the present invention. [Figure 1B] 2 is a schematic diagram illustrating a method flow for imaging according to an embodiment of the present invention. [Figure 2A] FIG. 4 is a schematic diagram illustrating another flow of a method for imaging according to an embodiment of the present invention. [Figure 2B] FIG. 4 is a schematic diagram illustrating another flow of a method for imaging according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a block diagram of an image fusion algorithm based on convolutional neural networks according to an embodiment of the present invention. [Figure 4A] FIG. 2 is a structural block diagram of an apparatus for imaging according to an embodiment of the present invention. [Figure 4B] 1 is an example embodiment of an apparatus for imaging according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a structural block diagram of a real-time imaging device according to an embodiment of the present invention; [Figure 6] 2 is a schematic diagram of a selectable pixel unit of a pixel array module of a real-time imaging device according to an embodiment of the present invention; FIG. [Figure 7] 2 is a schematic diagram of a pixel unit used in a pixel array module of a real-time imaging device according to an embodiment of the present invention. FIG. [Figure 8] 1 is a schematic diagram of a selectable architecture of a pixel array module of a real-time imaging device according to an embodiment of the present invention; [Figure 9] 1 is a multi-level shift circuit used in a row / column driving module of a real-time imaging device according to an embodiment of the present invention. [Figure 10] FIG. 2 is a schematic diagram of a selectable sequential logic scheme of a row / column decode module of a real-time imaging device according to an embodiment of the present invention. [Figure 11] FIG. 2 is a schematic diagram of a selectable combinatorial logic scheme of a row / column decode module of a real-time imaging device according to an embodiment of the present invention. [Figure 12] 1 is a schematic diagram of a readout module for a low-resolution, high-bitwidth original image of a real-time imaging device according to an embodiment of the present invention; FIG. [Figure 13] 1 is a schematic diagram of a readout module for high-resolution low-bit-width difference images of a real-time imaging device according to an embodiment of the present invention; FIG. [Figure 14] 1 is a schematic diagram of a readout module for a low-resolution, high-bitwidth original image and a high-resolution, low-bitwidth difference image of a real-time imaging device according to an embodiment of the present invention. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] Hereinafter, the method, apparatus and computer readable medium according to the present invention will be described in an exemplary manner with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part, not all, of the embodiments of the present disclosure. In general, the components of the embodiments of the present disclosure described and shown in the drawings may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the present disclosure sought to be protected, but is intended to only represent selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts are included within the protection scope of the present disclosure.

[0039] Note that in the following figures, like numbers and notations represent like items, and therefore, once an item is defined in one figure, it does not require further definition or explanation in subsequent figures.

[0040] 1A-1B are a schematic flow diagram of a method 100 for imaging according to an embodiment of the present invention, which may be used to perform image acquisition, processing, and output.

[0041] As shown in FIG. 1A, in the method 100, two images (e.g., two images with different resolutions and bit widths) are collected (or photographed) in parallel for a target field of view, i.e., a first image (e.g., a low-resolution, high-bit width original image) is obtained using a process or channel (step S10) shown in the right half of FIG. 1A, and a second image (e.g., a high-resolution, low-bit width difference image (obtained after difference processing of the corresponding original image)) is obtained using a process or channel (step S20) shown in the left half of FIG. 1A. When one image acquisition component is used, the collection of the two kinds of images is performed one after the other. When two image photographing components are used, the collection of the two kinds of images may be performed simultaneously, so-called in parallel. The present invention does not impose any limitation on the number of image photographing components used.

[0042] Then, the collected first image and the second image can be fused (step S30). According to an embodiment, the first image can be acquired with a first resolution and a first bit width, and the second image can be acquired with a second resolution and a second bit width, where the second image can be a difference image processed by difference processing. According to an embodiment, the first resolution can be lower than the second resolution, and the first bit width can be larger than the second bit width, for example, the first resolution can be much lower than the second resolution (for example, 10 times lower), the first bit width can be 8 bits, and the second bit width can be 2 bits. In this case, relatively speaking, the first image with a low resolution and a high bit width can be used to acquire rough information of the target field of view (for example, the background), and the second image with a high resolution and a low bit width can be used to acquire fine information of the target field of view (for example, the vehicles present in the target field of view), and the first image contains a small amount of data, and the second image contains a relatively large amount of data.

[0043] According to an embodiment of the present invention, the second image can be subjected to a difference process, which includes, for each captured pixel point (e.g., each pixel point in the second image captured at the second resolution), quantifying the difference between the pixel point and its adjacent or nearby pixel points and obtaining the quantified difference as the value of the corresponding pixel point in the second image, thereby obtaining a second image having a second bit width (e.g., a low bit width of 2 bits).

[0044] In the above and following description, the present invention does not impose any limitations on the so-called first or second resolution (or high / low resolution) and the first or second bit width (or high / low bit width), the meanings of which are relative, and those skilled in the art can set their specific values ​​as necessary to achieve advantageous technical effects.

[0045] Optionally, as shown in FIG. 1B, after the first and second images are collected (steps S10 and S20), the collected images are coded (steps S101 and S201) to improve the transmission efficiency during subsequent transmission (steps S102 and S202). After receiving the transmitted coded images, the receiving side performs decoding (steps S103 and S203) and image fusion (step S30) on the transmitted coded images, and finally obtains a third image (e.g., a high-resolution, high-bitwidth image of the field of view of interest). Here, the third image is a complete image of the target field of view, which can include the rough information of the first image and the fine information of the second image. Here, the complete image refers to an image of the complete area of ​​the field of view (e.g., a high-resolution, high-bitwidth image). At the receiving side, one receiver may alternately receive the coded data of the two types of images, or two receivers may simultaneously receive the coded data of the two types of images separately. Here, the present invention does not impose any restrictions on the receiving method.

[0046] 2A-2B are schematic diagrams showing another flow of a method 200 for imaging according to an embodiment of the present invention. The method 200 can be considered as an improvement of the method 100. As shown in FIG. 2A and FIG. 2B, similar to FIG. 1A-1B, the method 200 can also be used to collect two images in parallel, respectively, that is, a process or channel shown in the right half of the figure (step S10) is used to obtain a first image (e.g., a low-resolution, high-bit width original image), and a process or channel shown in the left half of the figure (step S20) is used to obtain a second image (e.g., a high-resolution, low-bit width difference image). Here, the same or similar steps are omitted from the description.

[0047] Unlike in Figs. 1A-1B, in Figs. 2A and 2B, before image fusion (step S30), an image recognition method can be used to identify (step 40) an area of ​​interest in the second image (and optionally, the first image, as shown by the dotted lines in the drawings, which may contain coarse information). For example, image recognition (e.g., for a specific object) and detection of motion features are performed for a target of interest in the field of view based on empirical data (e.g., morphological features of different targets), data of one image (e.g., previously taken) or multiple consecutive images, or multiple images taken within a predetermined time range. The image recognition method may only include identifying an image area where a specific object may be present, or may or may not include identifying a specific object (e.g., a vehicle) included in the area of ​​interest, but the present invention does not impose any restrictions on the specific image recognition method. The image recognition method may be any method or algorithm that can extract an area of ​​interest (or a moving area with motion) from an image.

[0048] Additionally, either before or after image fusion, artificial vision or computer scanning can be used to identify and dynamically distinguish objects in the field of view based on empirical data and information about the morphology of the objects of interest.

[0049] According to an embodiment of the present invention, the first image and the second image can be directly fused (to obtain a third image as a complete image of the field of view), or a portion of the first image can be fused with a corresponding portion of the second image (to obtain a third image as a partial image of the field of view). For example, fusing the first image and the second image can include determining a region of interest of the target field of view in the second image using an image recognition method, obtaining a corresponding region of the first image corresponding to the region of interest, and fusing the corresponding region of the first image with the region of interest in the second image to obtain the third image. For example, the third image can be obtained by fusing the region of interest with the corresponding region only if the region of interest includes a specific object (e.g., the region of interest is determined to include a specific object such as a car or a person by a method such as computer identification or manual identification).

[0050] Furthermore, the third image may be output after fusing, or the third image may be used to update the complete image and then the updated complete image may be output.

[0051] According to an embodiment of the present invention, a method (eg, method 100) for ultra-high resolution real-time imaging may include the following steps.

[0052] Step 1 (or step S10 and step S20): For the target field of view, capture and quantify, for example, obtain a high-resolution low-bit width (for example, 2 bits) difference image and a low-resolution high-bit width (for example, 8 bits) original image. By reducing the number of bits of the collected difference image (i.e., obtained by processing the difference between the original image when capturing), the collection speed is improved, which contributes to achieving the effect of real-time imaging. According to an embodiment of the present invention, for example, a 2-bit ternary bit width (for example, {-1, 0, 1}) difference image can be used for image fusion to obtain a clear fusion image. The collection of the low-resolution high-bit width original image can be realized using a normal commercially available sensor chip.

[0053] Step 2 (or step S101 and / or step S201): Using the data sparsity of the difference image, the high-resolution low-bit-width difference image is compression-encoded to reduce the data transmission bandwidth, thereby contributing to achieving the purpose of real-time transmission. Optionally, the low-resolution high-bit-width original image can also be compression-encoded as needed, and the data size of the low-resolution high-bit-width original image is usually much smaller than the data size of the high-resolution low-bit-width difference image.

[0054] For example, the compression coding method includes optimized run-length coding and Huffman coding. Run-length coding codes the original bit sequence into a count sequence L that records the number of times the data is repeated, and a data sequence D that records the repeated data itself. On the other hand, the count sequence L that records the number of times the data is repeated has a very uneven numerical distribution, so the amount of data can be further reduced by coding it using the Huffman variable-length coding method instead of the coding method using a fixed-length code. On the other hand, for the data sequence D that records the repeated data itself, theoretically, three different values ​​of the difference image should be expressed by two bits, but in the data sequence D of run-length coding, two adjacent data are not necessarily equal, so there are only two possibilities for the next data of the current data. Therefore, the larger of the possible values ​​is recorded as 1, and the smaller of the possible values ​​is recorded as 0. By coding based on the above principle, the amount of data can be further reduced, and the transmission bandwidth meets the conditions for real-time transmission (step S102 or step S202).

[0055] Step 3 (or step S103 or step S203): Decode the transmitted high-resolution, low-bitwidth difference image encoded in step 2 (and, optionally, the transmitted, encoded low-resolution, high-bitwidth original image).

[0056] Step 4 (or step S30): Fuse the (decoded) high-resolution low-bit-width difference image with the (decoded) low-resolution high-bit-width original image. In the present invention, an image fusion algorithm based on a convolutional neural network can be used to fuse and reconstruct them.

[0057] According to an embodiment of the present invention, when collecting a low-resolution, high-bit-width original image, the captured field of view is the target field of view of interest, while when capturing a high-resolution, low-bit-width difference image, it is ideal to use the same field of view as the low-resolution, high-bit-width original image. However, in practice, the two operations are not necessarily performed simultaneously, or even on the same device, so the fields of view targeted by the two images are not necessarily the same. According to the spirit of the solution of the present invention, as long as the field of view of the first image (e.g., the low-resolution, high-bit-width original image) is approximately the same as the field of view of the second image (e.g., the high-resolution, low-bit-width difference image), the third image generated by the method and device of the present invention can be obtained through subsequent image processing (e.g., image fusion that preserves the same parts of the two fields of view) to update the complete image or the region of interest of the field of view. In other words, according to an embodiment of the present invention, the above two fields of view corresponding to the first image and the second image can be called target fields of view, and the target field of view can be limited to a field of view in a specific time and / or a specific space, allowing slight errors in time and space.

[0058] According to an embodiment of the present invention, a method for obtaining a high-resolution low-bit-width difference image may be, for example, to capture one high-resolution original image in the same field of view as the original image, and compare the difference between a pixel point in the high-resolution original image and an adjacent or nearby pixel point of the pixel point. For example, the comparison result (difference between the two) can be quantified to one of {-1, 0, +1} based on the judgment of {smaller than, equal to, larger than} (this may be called the "three-section method"). The above operation (difference comparison and quantification) can be repeated for multiple pixel points of the high-resolution original image to finally generate a high-resolution low-bit-width difference image. For example, the above operation may be repeated for each pixel point, or may be repeated for pixel points at regular intervals (e.g., every two adjacent rows / columns) or variable intervals, or for multiple representative pixel points of the field of view selected by other methods, and the present invention does not impose any restrictions thereon. In this specification, the above-mentioned difference comparison and quantification process may be called "difference processing".

[0059] For example, the difference process may be to compare the values ​​of two adjacent pixel points, i.e., to compare the pixel point values ​​of the n+1th pixel and the nth pixel in a certain direction. In a similar manner, the pixel point value of the n+2th or n+3th pixel may be compared with the pixel point value of the nth pixel, i.e., the pixel point value of the n+ith pixel (i=1, 2, ...) may be compared with the pixel point value of the nth pixel. Such a case is particularly suitable for general situations that do not require high spatial resolution, such as, for example, general screening (e.g., whether there is a new object in the field of view, etc.), and can be used to detect whether there is an abnormality. For example, the difference comparison can be performed every two rows or every two columns.

[0060] As mentioned above, when the value of a pixel point is quantified to one of {-1, 0, +1}, the value of the pixel point may be, for example, a luminance value or other value representing a color, and this value may be, for example, one of 256 values, and may be represented by eight binary bits. When performing a difference comparison of the values ​​of the pixel points, the judgment can be based only on the direct difference between the two, that is, as long as the values ​​of the two pixel points are different, even if the difference corresponds to only one value of the 256 values ​​(for example, if the difference is 1), it is judged as -1 or 1, and accordingly, if the difference is 0, it is judged as 0. Note that the judgment criterion can be set so that the difference between the two is equal to or greater than a threshold value, for example, 4 out of 256 values, and if the judgment criterion is met, it is judged to be different and output -1 or 1 accordingly. This processing method can be used, for example, to qualitatively represent the details of edges in a high-contrast image.

[0061] In alternative embodiments of the invention, more sophisticated quantification procedures may be used, as described below.

[0062] For example, using the five-interval method, two thresholds are used, namely thresholds C1 and C2, where C1 and C2 are both greater than 0 and C2>C1; It is possible to determine whether the comparison result (difference) of the pixel point value is in the range of <-C2, <-C1, close to a reference value (e.g. 0), and satisfies the conditions of >C1, >C2. In other words, it is determined which range the comparison result is in, and {-2, -1, 0, 1, 2} is substituted for the comparison result of the pixel point value, that is, as follows.

number

[0063] The 4-interval method, which is slightly more resource-saving than the 5-interval method above, also uses the same two thresholds, i.e., thresholds C1 and C2, both of which are greater than 0, and C2>C1. It can be determined whether the comparison result of the pixel point value is in an interval close to <-C2, <-C1, or a reference value (e.g., 0), and >C1. In other words, it determines which interval the comparison result is in, and substitutes {-2, -1, 0,1} into the comparison result of the pixel point value, respectively, as follows:

number

number

[0064] Based on any one of the above three-section (three-value) method, four-section (four-value) method, and five-section (five-value) method, difference images of different bit widths can be obtained. Hereinafter, the present invention will be described in detail with reference to the attached drawings and specific three-value examples of the present invention, but the following examples are merely illustrative and not limiting. Note that the principle of the four-value or five-value examples is the same as the principle of the following three-value examples. EXAMPLES

[0065] In this embodiment, a method (e.g., method 100) for imaging is designed based on a high-resolution, low-bit-width (ternary, {-1, 0, 1}) difference image and a low-resolution, high-bit-width (e.g., 8-bit) grayscale image (e.g., method 100), and the specific steps are as follows:

[0066] Step 1-1 (or step S10 and step S20): Shoot and quantify the target field of view to obtain a low-resolution high-bit width grayscale image and a high-resolution low-bit width difference image. For example, the resolution of the low-resolution high-bit width grayscale image may be 1 / 64 of the resolution of the high-resolution low-bit width difference image, and the high-resolution low-bit width difference image may be based on the difference between two adjacent columns in the corresponding original image, and its default quantification interval may be, for example, [-255,-4], [-4,4], and (4,255]. In this way, the difference is respectively substituted with {-1,0,1} using the three-interval method according to the quantification interval (i.e., these three intervals correspond to -1,0,1, respectively). By reducing the number of bits of the difference image to be collected, the collection speed is improved, which contributes to achieving the effect of real-time imaging.

[0067] Step 1-2 (or steps S101, S102, and / or steps S201, S202): By utilizing the sparsity of the difference image, the high-resolution low-bitwidth difference image is compression-encoded to reduce the data transmission bandwidth, thereby contributing to achieving the purpose of real-time transmission.

[0068] For example, the compression encoding method consists of optimized run-length encoding and Huffman encoding. Specifically, in run-length encoding, the original bit sequence is encoded into a count sequence L that records the number of times the data is repeated, and a data sequence D that records the repeated data itself. On the other hand, the count sequence L that records the number of times the data is repeated has a very uneven numerical distribution, so the amount of data can be further reduced by encoding it using Huffman code instead of a fixed-length code encoding method. On the other hand, for the data sequence D that records the repeated data itself, theoretically, two adjacent pieces of data are not always equal. In this case, there are three possible values ​​for the difference image, and only two possible values ​​for the next data of the current data, so in the data sequence D, only the first piece of data has three possibilities, and all subsequent pieces of data have only two possibilities. Therefore, the first piece of data can be encoded using two bits, and each of the remaining pieces of data can be encoded with one bit.

[0069] Step 1-3 (or step S103 and / or S203): At the receiving end, the encoded high-resolution low-bitwidth difference image is decoded.

[0070] Step 1-4 (or step S30): Fuse the high-resolution, low-bit-width difference image and the low-resolution, high-bit-width grayscale image.

[0071] In this embodiment, an image fusion algorithm based on a convolutional neural network can be used to perform optimized fusion and reconstruction for both. In this embodiment, the input high-resolution low-bit width difference image and the input low-resolution high-bit width image both have one channel, and the resolution of the output high-resolution high-bit width image is the same as the resolution of the input high-resolution low-bit width difference image, and the number of channels is one.

[0072] 3 is a block diagram of an image fusion algorithm based on a convolutional neural network according to an embodiment of the present invention, the network structure of which is described below with reference to FIG.

[0073] According to an embodiment of the present invention, two input images (e.g., a high-resolution low-bit-width image and a low-resolution high-bit-width image) may have different resolutions, for example, the resolution of one image is 1 / 64 of the other image. To address this resolution mismatch issue, a multi-scale feature fusion network may be used to fuse high-frequency information from the high-resolution low-bit-width difference image and low-frequency information from the low-resolution high-resolution image at different scales.

[0074] As shown in Fig. 3, this multi-scale feature fusion network may be divided into three different branches, namely one super-resolution branch and two difference branches. The input of the super-resolution branch is a low-resolution high-bitwidth image, and the output is a corresponding (e.g., 8*8 times) high-resolution high-bitwidth image, and the latter half of the super-resolution branch uses feature fusion combination to fuse high-frequency components from the two difference branches to synthesize a high-resolution high-bitwidth image with clear details. The inputs of the two difference branches are high-resolution low-bitwidth difference images, but in this embodiment, since the difference image only has one x-direction (i.e., column-to-column difference comparison), the inputs of the two branches are the same, both of which are x-direction difference images. For better fusion, the two difference branches output high-resolution, high-bitwidth difference images in the x direction (i.e., column-to-column difference comparison) and y direction (i.e., row-to-row difference comparison), respectively. To achieve this, the supervision maps used by the two difference branches when training are high-resolution, high-bitwidth difference images in the x direction and y direction, respectively. Both of the two difference branches fuse low-frequency components at different scales from the super-resolution branch to guide the fusion of the high-resolution, high-bitwidth difference images.

[0075] The details of each branch structure, including feature fusion combinations, loss functions and training methods, are introduced below.

[0076] Referring to FIG. 3, the super-resolution branch can be divided into two parts. In the first part, the input low-resolution high-bitwidth image (e.g., 8*8 times) super-resolution processing is completed, and feature maps at different scales are obtained, which may be used to guide the fusion and reconstruction of the difference branch. In the second part, the feature maps from the difference branch are fused to complete the fusion and reconstruction of the final high-resolution high-bitwidth image. The super-resolution branch may use a progressive super-resolution algorithm, for example, a 2*2 upsampling layer can be used three times to achieve 8*8 times super-resolution, and the upsampling layer is realized by transposed convolution. In the second half of the network, the super-resolution branch fuses the features of the two difference branches to complete the final fusion and reconstruction.

[0077] Referring to FIG. 3, the two differential branches have the same structure, but their supervision maps may be different and may not share parameters. The first half of the differential branch may use a U-shaped structure similar to U-net, but unlike U-net, RRDB (Residual-in-Residual Dense Block) may be used to replace the basic convolution layer. A max pooling layer may be used as the downsampling layer, and a transposed convolution layer may be used as the upsampling layer. The RRDB and the max pooling layer may be combined as the basic downsampling module, and in the downsampling process, a 2*2 downsampling module may be used three times to obtain four feature maps at different scales (e.g., the original scale, 1 / 2*1 / 2 times resolution, 1 / 4*1 / 4 times resolution, and 1 / 8*1 / 8 times resolution), and these feature maps are fused with the feature maps in the upsampling process in the manner shown in FIG. 3. The RRDB and the transposed convolution layer are combined as a basic 2*2 upsampling module, and the 2*2 upsampling module is used three times to restore the feature map to its original size. In the upsampling process, the feature map obtained in the downsampling of the same differential branch and the feature map obtained in the upsampling process of the super-resolution branch can be fused at different scales.

[0078] According to an embodiment of the present invention, the feature fusion combination algorithm includes three kinds of feature fusion combinations, namely, high-resolution difference image to high-resolution difference image (HRD-to-HRD) feature fusion combination, low-resolution original image to high-resolution difference image (LRI-to-HRD) feature fusion combination, and high-resolution difference image to low-resolution original image (HRD-to-LRI) feature fusion combination, which are respectively located at different positions in the network and play different roles, and each feature combination performs a concatenation operation (Concat) of feature maps. In the first half of the entire network, the three branches are all multi-scale structures, and in this part, the feature maps of the super-resolution branch can be fused to the upsampling parts of the two difference branches (LRI-to-HRD feature fusion combination), and the feature maps generated in the downsampling process of the same difference network can be fused to the upsampling part (HRD-to-HRD feature fusion combination). In the second half of the entire network, the feature maps generated by the two difference branches can be fused to the super-resolution branch (HRD-to-LRI feature fusion combination) to complete the final fusion. In this network, the number of feature maps in each layer is set to 16, except for the layer containing feature fusion connections, in which the number of feature maps is an integer multiple of 16, where the multiple is the number from different branches that are fused, e.g., when fusing two branches, the number of feature maps is 32.

[0079] According to an embodiment of the present invention, the minimum mean square error between the clear high-resolution high-bit width image and the fusion result may be used as the loss function (MSELoss) of the super-resolution branch. In addition, the minimum mean square error between the high-resolution high-bit width difference images in the x and y directions and the outputs of the two difference branches may be used as the loss functions of the two difference branches. Therefore, the total loss function can be expressed as the following formula:

number

[0080] According to an embodiment of the present invention, for the training method, the DIV2K super-resolution dataset can be used to create a training set. For example, during training, the convolution kernel size is set to 3, the Adam algorithm is used as the optimizer, the hyperparameters β and γ in the loss function are set to 0.1, and the learning rate is set to 1×10 -4 , multiply the learning rate by a decay factor of 0.5 every 20K iterations, train for a total of 100K iterations, and set the batchsize to 16. EXAMPLES

[0081] This embodiment proposes a method for imaging with added image recognition based on a difference image, such as the method 200. Unlike performing image recognition directly on the original image without performing subtraction processing, this embodiment performs image recognition directly on the difference image, identifies the region of interest, and then fuses only the region of interest.

[0082] Step 2-1: Same as step 1-1 in Example 1. Step 2-2: Same as step 1-2 in Example 1. Step 2-3: Same as step 1-3 in Example 1. Step 2-4: Construct a training dataset, and use the generated training set to train an image recognition method (e.g., YOLOv3) to obtain a trained image recognition method. The training dataset in this embodiment may be generated by a public image recognition dataset or by manual labeling. The two methods are specifically as follows: 1) Generate from a public dataset: After downloading a public image recognition dataset, do not process the labels, but perform the above-mentioned difference processing on the original images in the dataset (e.g., unquantified high-resolution original images) based on the quantification interval set in step 2-1, obtain the corresponding difference images (e.g., high-resolution low-bitwidth difference images), and combine them with the existing labels in the dataset to generate a paired training set. 2) Generation by manual labeling: Regions of interest in the collected difference images are manually labeled to form data pairs, thereby building a training dataset. Here, step 2-4 may be generally performed in advance, i.e., an existing trained image recognition method may be adopted when implementing this method. Generally, after obtaining a high-resolution low-bitwidth difference image in step 2-3, we proceed directly to step 2-5. Step 2-5: The trained image recognition method (YOLOv3 obtained in step 2-4) may be applied to the high-resolution low-bitwidth difference image to identify regions of interest. Step 2-6: Obtain a corresponding region of the low-resolution high-bitwidth image that corresponds to the identified region of interest (e.g., as determined by positioning the region of interest in the target field of view). Step 2-7: Fuse the identified region of interest and the identified corresponding region to obtain a fused image (third image) for the region of interest, where the fusion method may be the same as step 1-4 in embodiment 1. For example, the fused image may be used to update a complete image of the field of view to obtain an updated complete image. The complete image may be updated (for example, using the fused image) only when a change occurs in the field of view (for example, a specific object appears, the original object moves, etc.). Alternatively, different update rates may be set, for example, the first image and the second image (images containing information of the entire field of view) are used to update the complete image at the first update rate, and the fused image is used to update the complete image (corresponding parts that need to be updated or where changes have occurred) at the second update rate, where the first update rate may be smaller than the second update rate (for example, 1 fps and 30 fps, respectively), and the present invention does not impose any restrictions on their specific values. In this way, the complete image can be updated with the least amount of data. EXAMPLES

[0083] This embodiment proposes a means for designing a method for imaging that can self-adaptively adjust the quantification interval based on a high-resolution, low-bit-width (e.g., 3-value) difference image and a low-resolution, high-bit-width (e.g., 8-bit) grayscale image. Step 3-1: Construct a database of code rate-quantification interval-fusion quality, that is, construct a database of code rate-quantification interval-fusion quality using an existing public dataset. Step 3-2: Determine the quantification interval for collecting low-resolution grayscale images and high-resolution difference images. Set the initial code rate and fusion quality, for example, the resolution of the low-resolution grayscale image is set to 1 / 64 of the high-resolution difference image, and the high-resolution difference image is set to be based on the difference between two adjacent columns of its corresponding original image. Based on the set initial system bit rate and fusion quality, query the database built in step 3-1 to determine the corresponding quantification interval, complete the collection of low-resolution grayscale images and high-resolution difference images, and give them different bit widths. The code rate and fusion quality can be manually adjusted in real time, and the corresponding quantification interval in the database is also changed accordingly. Next, steps 1-1 to 1-4 may be performed, and steps 2-1 to 2-7 may be performed. EXAMPLES

[0084] This embodiment proposes a method (e.g., method 200) for imaging with added motion detection based on difference images. Unlike conventional motion detection on fused images (e.g., third image), the method of this embodiment performs motion detection (e.g., image recognition shown in step S40 of Fig. 2A-2B) on images before fusion (e.g., low-resolution high-bit width image or first image, and / or high-resolution low-bit width image or second image, as described above). After identifying the motion area (or region of interest), fusion is performed only on the motion area (or region of interest) (e.g., image fusion shown in step S30 of Fig. 2A-2B). The method can be implemented with reference to Fig. 2A-2B. Hereinafter, specific steps of the method will be described by taking a high-resolution low-bit width difference image as an example.

[0085] Step 4-1: Same as step 1-1 in Example 1. Step 4-2: Same as step 1-2 in Example 1. Step 4-3: Same as step 1-3 in Example 1. Step 4-4: Using an inter-frame method, calculate the difference between the high-resolution low-bitwidth difference images of two adjacent frames, extract areas with changes in the difference (e.g., the difference meets a certain threshold condition), and obtain the motion area (e.g., in the form of coordinates), i.e., the region of interest. Step 4-5: Same as steps 2-6 to 2-7 in Example 2, and obtain the fused motion region. Step 4-6: The fused motion regions are used to update a previously captured full image of the target field of view, and regions other than the motion regions in the full image are not updated. Step 4-7: Repeat steps 4-1 to 4-6. EXAMPLES

[0086] This embodiment proposes a method for imaging based on a low-resolution, high-bit-width (eg, 8-bit) RGB image and a high-resolution, low-bit-width (eg, ternary) difference image.

[0087] Step 5-1: Collect a low-resolution high-bit-width RGB image and a high-resolution low-bit-width difference image. In this example, the resolution of the low-resolution high-bit-width RGB image is 1 / 64 of the high-resolution low-bit-width difference image, and the high-resolution low-bit-width difference image can be based on the difference between two adjacent columns of the corresponding original grayscale image, whose default quantification intervals are (-255,-4), [-4,4], and (4,255), and the corresponding quantification values ​​are -1, 0, and 1. By reducing the number of bits of the difference image to be collected, it is possible to improve the collection speed and achieve the effect of real-time imaging. Step 5-2: Same as step 1-2 in Example 1. Step 5-3: Same as step 1-3 in Example 1. Step 5-4: Combine and fuse the collected low-resolution high-bit-width RGB color images. In this embodiment, an image fusion algorithm based on a convolutional neural network is used to perform optimized fusion and reconstruction for both of them. In this embodiment, for the convolutional neural network, the channel numbers of the input high-resolution low-bit-width difference image and the low-resolution high-bit-width RGB image are 1 and 3 respectively, and the resolution of the output high-resolution high-bit-width image is the same as the resolution of the input high-resolution low-bit-width difference image, and the channel number is 3, that is, a high-resolution high-bit-width RGB image is output.

[0088] Embodiments of an apparatus for real-time imaging according to the present invention FIG. 4A is a structural block diagram of an apparatus 400 for imaging according to an embodiment of the present invention, and FIG. 4B is an example embodiment of the apparatus 400 for imaging according to an embodiment of the present invention.

[0089] As shown in FIG. 4A, the device 400 may include an image capture component 401, a data processing component 402, an encoding component 403, and a transmission component 404. According to an embodiment, the image capture component 401 may be configured to perform an operation of acquiring an image (including capturing and quantifying), as described above. The data processing component 402 may be configured to perform an operation related to data processing, including operations such as image recognition, image fusion, and decoding. The encoding component 403 may be configured to encode data of the acquired image, for example, for transmission to the data processing component 402 or other data processing devices. The transmission component 404 may be configured to transmit various data, for example, transmitting encoded image data from the encoding component 403 to the data processing component 402 for further processing. The various components included in the device 400 may perform various operations, for example, based on the control of a control component (not shown).

[0090] As shown in Fig. 4B, an apparatus (e.g., apparatus 400) according to the present invention may be implemented to include a real-time imaging device demo board 410 (which can be used as a combination of image capture component 401 and encoding component 403) and a host computer 412 (which can be used as a data processing component 402), and the two may be electrically connected via an Ethernet interface 414 (which can be used as a transmission component 404). The real-time imaging device demo board 410 may include a real-time imaging device 411, an FPGA control unit 415, and a system peripheral chip 413. The host computer 412 uses the Ethernet interface 414 to configure the real-time imaging device demo board 410, and the FPGA control unit 415 and the system peripheral chip 413 in the real-time imaging device demo board 410 provide the timing signals and control voltages required for operation to the real-time imaging device 411 according to the configuration information, and finally transmit the collected image data to the host computer 412 via the Ethernet interface 414 to complete one collection task.

[0091] Figure 5 is a structural block diagram of a real-time imaging device (e.g., real-time imaging device 411) according to an embodiment of the present invention. As shown in Figure 5, the real-time imaging device can include a pixel array module, a row / column driving module, a row / column decode module, a readout module, and an I / O module. An FPGA control unit (e.g., FPGA control unit 415) controls the row / column decode module through the I / O interface module, provides working voltage to the row / column driving module, and controls the pixel array module to perform light sensing for a target scene, and finally reads out image data by the readout module.

[0092] 6 is a schematic diagram of selectable pixel units of a pixel array module of a real-time imaging device (e.g., real-time imaging device 411) according to an embodiment of the present invention. As shown in FIG. 6, the pixel array module can include two levels: pixel unit and pixel array. At the pixel unit level, there are many options, such as photodiode (PD), phototransistor (PT), charge-coupled device (CCD), active pixel sensor (APS), etc., based on planar silicon process manufacturing technology.

[0093] FIG. 7 is a schematic diagram of a pixel unit used in a pixel array module of a real-time imaging device (for example, real-time imaging device 411) according to an embodiment of the present invention. FIG. 8 is a schematic diagram of a selectable architecture of a pixel array module of a real-time imaging device (for example, real-time imaging device 411) according to an embodiment of the present invention. In the present invention, in one preferred embodiment of the pixel unit shown in FIG. 6, the dual transistor photodetector in CN201210442007.X shown in FIG. 7 can be used. As shown in FIG. 8, the pixel array level can select NAND architecture or NOR architecture, and in one preferred embodiment, the NOR architecture can be used.

[0094] 9 is a multi-level shift circuit used in a row / column driving module of a real-time imaging device (such as the real-time imaging device 411) according to an embodiment of the present invention. As shown in FIG. 9, as an example, the row / column driving module can use the multi-level shift circuit in CN202010384765.5. The module needs to input a pre-shift positive voltage signal VVPP, a pre-shift negative voltage signal VVPN, a shift positive voltage signal VPHV, a shift negative voltage signal VNHV, and a pre-shift control signal VIN. The shift voltage output signal VO outputs the shift positive voltage signal VPHV or the shift negative voltage signal VNHV under the control of the pre-shift control signal VIN to provide a driving voltage for the pixel array module.

[0095] According to an embodiment of the present invention, the row / column decode module can use a sequential logic scheme of a shift register or a combinatorial logic scheme of a decoder. Figure 10 is a schematic diagram of a selectable sequential logic scheme of a row / column decode module of a real-time imaging device (e.g., real-time imaging device 411) according to an embodiment of the present invention. In this embodiment, a combinatorial logic scheme written in Verilog HDL is used. Figure 11 is a schematic diagram of a selectable combinatorial logic scheme of a row / column decode module of a real-time imaging device (e.g., real-time imaging device 411) according to an embodiment of the present invention. As an example, a 1024-bit sequential logic scheme (Figure 10) and an 8-bit combinatorial logic scheme (Figure 11) can be used.

[0096] 12-14 are schematic diagrams of different embodiments of the readout module of a real-time imaging device (e.g., real-time imaging device 411) according to an embodiment of the present invention. The readout module is implemented as two schemes, that is, readout circuits may be designed for high-resolution low-bit width difference images and low-resolution high-bit width original images, respectively, and readout circuits for high-resolution low-bit width difference images and low-resolution high-bit width original images may be combined. For the first scheme, the readout circuit in the existing patent CN201911257219.9 (as shown in FIG. 12) can be used to read out the low-resolution high-bit width original image, and the current subtraction circuit in the existing patent CN202010697791.3 (as shown in FIG. 13) can be used to read out the high-resolution low-bit width difference image. For the second scheme, the readout circuit in Patent CN201911257219.9 (as shown in FIG. 12 ) is subsequently used to read out a low-resolution, high-bit-width original image, the up-down counter is set to the up-counting mode, and when reading out a high-resolution, low-bit-width difference image, the up-down counter is first set to the up-counting mode and then set to the down-counting mode to realize the readout of the difference. The present invention uses a novel read-out circuit (as shown in FIG. 14), when working on a low-resolution, high-bit-width original image, DIR is 0, current mirror CM1 works, CM2 does not work, and the current of BLN is used to discharge into capacitor C, and the counter continues to work until the capacitor voltage is discharged below the reference voltage VP of comparator CMP1, and when the capacitor voltage is discharged below the reference voltage VP, the counter stops working and the quantification result is sent out by the parallel-serial conversion module; when working on a high-resolution, low-bit-width difference image, DIR is 1, current mirrors CM1 and CM2 work simultaneously, and the current of BLN-BLN+1 is used to discharge into capacitor C, and after passing through comparators CMP1 and CMP2, a 2-bit quantification result is obtained, and the result is sent out by the parallel-serial conversion module.The I / O interface module can use any manufacturer's generic input / output interface with ESD protection function.

[0097] In addition, each embodiment in this specification will be described step by step, focusing on the differences from other embodiments, and the same and similar parts between the embodiments may be mutually referred to.

[0098] In some embodiments provided in the present application, each clock in a flowchart or block diagram may represent a module, a program segment, or a portion of code, and the module, program segment, or portion of code may include one or more executable instructions for implementing a specified logical function. It should be understood that in some alternative implementations, the functions shown in the blocks may be executed in a different order than that shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, or may be executed in reverse order depending on the functionality. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be realized by a dedicated hardware-based system that executes the specified functions or operations, or may be realized by a combination of dedicated hardware and computer instructions.

[0099] When the function is realized in the form of a software functional module and sold or used as an independent product, it may be recorded in a computer-readable storage medium. Based on this understanding, the technical means of the present disclosure may be embodied in the form of a software product, either essentially or in part contributing to the prior art. The computer software product includes some instructions recorded in a storage medium to cause a computer device (such as a personal computer, a server, or a network device) to execute all or part of the steps of the method according to each embodiment of the present disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB memory, a portable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0100] It should be noted that, in this specification, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that such an actual relationship or sequence exists between these entities or operations. Furthermore, the terms "comprises," "has," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or device that includes a set of elements includes not only those elements, but also other elements not expressly listed, or includes the inherent elements of such a process, method, article, or device. In the absence of further limitations, an element limited by the statement "comprises..." does not exclude the presence of other identical elements in the process, method, article, or device that includes the element.

[0101] Based on the above disclosure of the present invention, it is not difficult for a person skilled in the art to reasonably understand the solutions of the present invention, such as the method, apparatus, etc., and the details not described in detail in the embodiments thereof, all of which are within the protection scope of the present invention.

Claims

1. acquiring a first image of a target field of view at a first resolution and quantifying the first image having a first bit width; a step of capturing an image of the target field of view at a second resolution and performing a difference process to obtain a second image having a second bit width, the difference process including, for a pixel point captured at the second resolution, quantifying a difference between the pixel point and an adjacent or nearby pixel point of the pixel point, and obtaining the quantified difference as a value of the corresponding pixel point in the second image; fusing the first image and the second image to obtain a third image; A method for imaging, wherein the first resolution is lower than the second resolution and the first bit width is greater than the second bit width.

2. The step of fusing the first image and the second image to obtain a third image includes: determining a region of interest of the target field of view in the second image using image recognition methods; obtaining a corresponding region in a first image that corresponds to the region of interest; and fusing the corresponding region of the first image with the region of interest of the second image to obtain a third image.

3. 10. The method for imaging of claim 1, further comprising the steps of encoding and transmitting a second image prior to fusing, and after said transmission, decoding the second image for use in fusing.

4. 4. The method for imaging of claim 3, wherein the second image is encoded using a run-length encoding scheme, and the bit sequence of the second image is encoded into a count sequence L that records the number of repetitions of the repeating data, and a data sequence D that records the repeating data itself.

5. 5. The method for imaging of claim 4, further comprising: recording said count sequence L using a Huffman code; and recording said data sequence D using a fixed length code.

6. 10. The method for imaging of claim 1, wherein fusing the first image and the second image comprises fusing the first image and the second image using a convolutional neural network.

7. 3. The method for imaging of claim 2, wherein determining the region of interest of the target field of view in the second image using image recognition methods includes determining the region of interest by combining images taken of the target field of view within a specific time range or previously stored images taken of the target field of view.

8. 3. The method for imaging of claim 2, further comprising the step of training the image recognition method using deep learning based on selected targets, spatial conditions when photographing, and manual labeling.

9. The method for imaging of claim 2, wherein the step of fusing the region of interest and the corresponding region to obtain a third image further comprises the step of fusing the region of interest and the corresponding region to obtain a third image only if the region of interest includes a particular object.

10. 10. The method for imaging of claim 1, further comprising the step of outputting a third image after fusing, or outputting an updated complete image after updating the complete image with the third image.

11. an image capture component configured to capture and quantify an image of a target field of view at a first resolution to obtain an image having a first bit width, and capture and differentially process the target field of view at a second resolution to obtain a second image having a second bit width, the differential process including, for a pixel point captured at the second resolution, quantifying a difference between the pixel point and an adjacent or nearby pixel point of the pixel point, and obtaining the quantified difference as a value of the corresponding pixel point in the second image; a data processing component coupled to the image capture component and configured to fuse the first image and the second image to obtain a third image; An apparatus for imaging, wherein the first resolution is lower than the second resolution and the first bit width is greater than the second bit width.

12. Fusing the first image and the second image to obtain a third image includes: determining a region of interest of the target field of view in the second image using image recognition methods; obtaining a corresponding region in a first image that corresponds to the region of interest; 12. The apparatus for imaging of claim 11, further comprising fusing the corresponding region of the first image with the region of interest of the second image to obtain a third image.

13. further comprising an encoding component and a transmitting component; the encoding component is coupled to the image capture component and configured to encode a second image prior to fusing; the transmission component is coupled to the encoding component and to the data processing component and configured to transmit an encoded second image to the data processing component; The apparatus for imaging of claim 11 , wherein the data processing component is further configured to, after receiving an encoded second image, decode the encoded second image for use in fusion.

14. 14. The apparatus for imaging of claim 13, wherein the encoding component is further configured to encode the second image using a run-length encoding scheme to encode a bit sequence of the second image into a count sequence L that records a number of repetitions of the repeating data and a data sequence D that records the repeating data itself.

15. The apparatus for imaging of claim 14 , wherein the encoding component is further configured to record the count sequence L using a Huffman code and record the data sequence D using a fixed length code.

16. 12. The apparatus for imaging of claim 11, wherein fusing the first image and the second image comprises fusing the first image and the second image using a convolutional neural network.

17. 13. The apparatus for imaging of claim 12, wherein determining the region of interest of the target field of view in the second image using image recognition methods includes combining images taken of the target field of view within a specific time range or previously stored images taken of the target field of view to determine the region of interest.

18. 13. The apparatus for imaging of claim 12, wherein the data processing component is further configured to train the image recognition method using deep learning based on selected targets, spatial conditions when photographing, and manual labeling.

19. A non-transitory computer readable medium having recorded thereon program code which, when executed by a computer, performs the method of claim 1.

Images