Image fusion method and device
By detecting and deforming feature points in image frames, combining image frames to generate combined images, the problem of difficulty in object recognition in low-brightness environments is solved, brightness improvement and motion blur reduction are achieved, and recognition accuracy is improved.
Patent Information
- Application Number
- CN201811351115.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-11-14
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2038-11-14
AI Technical Summary
When shooting images in low-brightness environments, it is difficult to clearly identify objects. Existing methods such as increasing camera exposure time may lead to motion blur, and image processing methods are limited.
By detecting feature points in image frames, image deformation is performed based on these feature points and predefined reference points, image frames are combined to generate combined images, reducing motion blur and increasing brightness.
Generate combined images with increased brightness and minimized motion blur in low brightness environments, which are suitable for object recognition systems and improve recognition accuracy.
Smart Images

Figure CN110136224B_ABST
Abstract
Description
[0001] This application claims priority from Korean Patent Application No. 10-2018-0016451 filed on February 9, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field
[0002] The following description relates to an image processing technique. Background Art
[0003] When capturing an image in a low-brightness environment, it may be difficult to clearly identify an object in the captured image. As methods for solving this problem, there is a method of capturing an image by increasing the exposure time of a camera and a method of performing image processing (e.g., adjusting the brightness of an image captured by a camera). Both methods can increase the brightness of an image. However, the former method may cause motion blur in the image when the object moves, and the latter method is limited in clearly displaying the object. Summary of the invention
[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0005] In one general aspect, an image fusion method includes: detecting first feature points of an object shown in a first image frame from a first image frame, and converting the first image frame based on the detected first feature points and a predefined reference point; detecting second feature points of the object shown in the second image frame from a second image frame, and converting the second image frame based on the detected second feature points and the reference point; and obtaining a combined image by combining the converted first image frame and the converted second image frame.
[0006] The step of converting the first image frame may include converting the first image frame so that the first feature point is located at the reference point, and the step of converting the second image frame may include converting the second image frame so that the second feature point is located at the reference point.
[0007] The converting of the first image frame may include detecting a first object region from the first image frame based on the detected first feature points, and converting the first object region through image deformation based on the first feature points and a reference point included in the detected first object region.
[0008] The converting of the second image frame may include detecting a second object region from the second image frame based on the detected second feature points, and converting the second object region through image deformation based on the second feature points and reference points included in the detected second object region.
[0009] The obtaining may include combining the converted first object region and the converted second object region.
[0010] The obtaining step may include generating a combined image by combining first pixel values of the converted first image frame and second pixel values of the converted second image frame.
[0011] The image fusion method may further include: measuring the image quality of the first image frame or the converted first image frame; in response to the measured image quality satisfying a preset condition, determining to detect a first feature point from the first image frame, or determining to obtain a combined image based on the converted first image frame.
[0012] The image fusion method may further include detecting third feature points of an object shown in the third image frame from the third image frame, converting the third image frame based on the detected third feature points and a reference point, and combining the obtained combined image with the converted third image frame.
[0013] In another general aspect, a non-transitory computer-readable storage medium storing instructions is provided. When the instructions are executed by a processor, the processor is caused to perform the above-mentioned image fusion method.
[0014] In another general aspect, an image fusion device includes: a processor, which can be configured to: detect first feature points of an object shown in the first image frame from a first image frame; convert the first image frame based on the detected first feature points and a predefined reference point; detect second feature points of the object shown in the second image frame from a second image frame; convert the second image frame based on the detected second feature points and the reference point; and obtain a combined image by combining the converted first image frame and the converted second image frame.
[0015] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 An example of the overall operation of the image fusion device is shown.
[0017] Figure 2 is a flowchart illustrating an example of an image fusion method.
[0018] Figure 3 and Figure 4 is a flowchart illustrating an example of an image fusion method.
[0019] Figure 5 is a flowchart illustrating an example of applying the image fusion method to face verification.
[0020] Figure 6 and Figure 7 An example of performing image fusion based on a plurality of image frames is shown.
[0021] Figure 8 An example of the configuration of an image fusion device is shown.
[0022] Fig. 9 An example of a configuration of a computing device is shown.
[0023] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to represent the same elements, features and structures. The drawings may not be drawn to scale, and the relative sizes, proportions and descriptions of the elements in the drawings may be exaggerated for clarity, illustration and convenience. DETAILED DESCRIPTION
[0024] The following structural or functional descriptions are exemplary to describe only examples, and the scope of the examples is not limited to the descriptions provided in this specification. Various changes and modifications may be made thereto by those of ordinary skill in the art.
[0025] Although the terms "first" or "second" are used to explain various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another. For example, within the scope of the rights according to the concept of the present disclosure, a "first" component may be referred to as a "second" component, or similarly, a "second" component may be referred to as a "first" component. It will be understood that when a component is referred to as being "connected to" another component, the component may be directly connected or coupled to the other component, or there may be intermediate components.
[0026] As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It will also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of the stated features, integers, steps, operations, elements, components or combinations thereof, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0027] Unless otherwise defined herein, all terms (including technical or scientific terms) used herein have the same meaning as those generally understood by those of ordinary skill in the art. Unless otherwise defined herein, terms defined in general dictionaries should be interpreted as having a meaning that matches the contextual meaning in the relevant field and should not be interpreted as an idealized or overly formal meaning.
[0028] Hereinafter, examples will be described in detail with reference to the accompanying drawings, wherein the same drawing reference numerals are used for the same elements and thus repeated descriptions thereof will be omitted for the sake of brevity.
[0029] Figure 1An example of the overall operation of the image fusion device is shown.
[0030] Reference Figure 1 , the image fusion device 100 is a device that generates combined images 122, 124, 126, ... 128 by fusing at least two image frames 110 using a processor. For example, the image fusion device 100 generates a combined image by fusing a current image frame and n previous image frames that are temporally adjacent to the current image frame, where n is an integer greater than 0. For example, when it is assumed that the current image frame is image frame 116 and the image fusion device 100 is to generate a combined image based on three image frames, the image fusion device 100 may generate a combined picture by fusing image frames 112, 114, and 116. If the image fusion device 100 is to generate a combined image based on two image frames, the image fusion device 100 may generate a combined image by fusing image frame 116 as the current image frame and image frame 114 as the previous image frame.
[0031] The image frame 110 includes image frames 112, 114, 116, ..., 118 captured continuously in time by a camera. Here, the term "image frame" may be replaced by "input image" or "image". In the case where the image frame is captured in a dark environment with low brightness, it is difficult to clearly identify the object in the image frame. In application fields where object recognition is important, such as face verification, it is important to clearly identify the face area in the image frame even though the image frame is captured in a dark environment. The clear recognition of the face area greatly affects the accuracy of face verification. As a general solution to overcome the disadvantage of capturing image frames in a low-brightness environment, a method of capturing image frames by increasing the camera exposure time is adopted. However, when the exposure time increases, the possibility of causing motion blur in the image frame when the camera and / or the object moves increases, which again hinders object recognition.
[0032] In one example, by combining image frames taken in a low-brightness environment, the image fusion device 100 generates a combined image in which the brightness is improved and the object is relatively easily recognized. In this example, the image fusion device 100 does not simply combine the image frames, but converts the image frames based on the feature points of each frame of the image frames, and combines the converted image frames, thereby minimizing the occurrence of motion blur. For example, the image fusion device 100 extracts feature points representing the shape features of the object from the image frames, converts the image frames so that the extracted feature points are placed at the position of the reference point, and combines the converted images, thereby generating a combined image in which the brightness is improved and the motion blur is minimized. The image fusion device 100 is applicable to user authentication systems, financial transaction authentication systems, access control and monitoring systems, and object recognition systems using biometrics.
[0033] Hereinafter, the operation of the image fusion apparatus 100 will be further described with reference to the accompanying drawings.
[0034] Figure 2 is a flowchart illustrating an example of an image fusion method.
[0035] According to Figure 2 The operations of the image fusion method may be performed in the order shown in the figure, or some operations may be omitted, or the operations may be performed in a different order without departing from the technical spirit and scope of the examples described below. In addition, the operations may be performed in parallel or simultaneously. Figure 2 For example, operations 210, 220 and operations 230, 240 may be performed in parallel.
[0036] In operation 210, the image fusion device detects first feature points of an object shown in the first image frame from the first image frame. Different feature points are extracted based on the type of the object. For example, in the case where the object is a face, facial key points (e.g., two end points of the eye, the center of the eye, an end point of the nose, and two end points of the mouth) are extracted from the first image frame. In the case where the object is an object, feature points representing shape features of the corresponding object are extracted from the first image frame. For example, in the case where the object is a license plate of a vehicle, four vertices of the rectangular license plate are extracted as feature points.
[0037] In operation 220, the image fusion apparatus converts the first image frame based on the detected first feature point and the predefined reference point. Here, the reference point has a pre-fixed position for the type of the object, regardless of the position of the feature point extracted from the first image frame. For example, in the case where the object is a face, the position of the reference point is a reference position fixed at a facial part (e.g., eyes, nose, and mouth).
[0038] The image fusion device transforms the first image frame using image deformation (warp) so that the first feature point is placed at the position of the reference point. For example, the image fusion device calculates an affine transformation matrix indicating the correspondence between the first feature point and the reference point based on the position value of the first feature point and the position value of the reference point, and transforms the first image frame by applying the calculated affine transformation matrix to the position value of the pixel of the first image frame. In another example, the process of transforming the first image frame includes such a process: detecting a first object area (e.g., a face area) from the first image frame based on the first feature point, transforming the first object area by image deformation so that the first feature point contained in the detected first object area is placed at the position of the reference point.
[0039] In operation 230 , the image fusion apparatus detects second feature points (eg, facial key points) of the object shown in the second image frame from the second image frame in the same or similar manner as described in operation 210 .
[0040] In operation 240, the image fusion device converts the second image frame based on the second feature point and the reference point in the same or similar manner as described in operation 220. For example, the image fusion device converts the second image frame using an affine transformation matrix so that the second feature point is placed at the position of the reference point. In another example, the process of converting the second image frame includes a process of detecting a second object area from the second image frame based on the detected second feature point, and converting the second object area by image deformation so that the second feature point contained in the detected second object area is placed at the position of the reference point. Here, the position of the reference point is the same as the position of the reference point used in operation 220.
[0041] In operation 250, the image fusion device obtains a combined image by combining the converted first image frame and the converted second image frame. For example, the processing includes: combining the converted first object area of the first image frame and the converted second image area of the second image frame. The image fusion device generates the combined image by combining the first pixel value of the converted first image frame and the second pixel value of the converted second image frame. For example, the combination of the first pixel value and the second pixel value includes: the sum, average value, weighted sum or weighted average value of the first pixel value and the second pixel value at corresponding positions in the converted first image frame and the converted second image frame. However, the example is not limited to this.
[0042] In another example, the image quality of the image frame is considered to select the image frame for image fusion. The image fusion device measures the image quality of the input first image frame or the converted first image frame, and in response to the measured image quality meeting the preset condition, determines to detect the first feature point from the first image frame or determines to obtain the combined image based on the converted first image frame. For example, the image fusion device measures the brightness of the first image frame, and in response to the measured brightness of the first image frame being less than a preset threshold, determines to detect the first feature point from the first image frame or determines to obtain the combined image based on the converted first image frame. In response to the brightness of the first image frame being greater than or equal to the threshold, the image fusion device considers that the corresponding first image frame is not acquired in a low-brightness environment, and does not use the first image frame to acquire the combined image. Similarly, the image fusion device measures the image quality (e.g., brightness) of the input second image frame or the converted second image frame, and in response to the measured image quality meeting the preset condition, determines to detect the second feature point from the second image frame or determines to obtain the combined image based on the converted second image frame. For example, in response to the brightness of the second image frame being greater than or equal to the threshold, the image fusion device does not use the corresponding second image frame to obtain the combined image.
[0043] In operation 260, the image fusion device converts the combined image. In one example, operation 260 is selectively performed. The image fusion device converts the combined image based on the correspondence between the second feature point of the second image frame as the most recently input image frame and the reference point. When it is assumed that the affine transformation matrix is determined based on the position of the second feature point and the position of the reference point, and the second image frame is converted by the corresponding affine transformation matrix in operation 240, the image fusion device restores the shape of the object shown in the combined image to its original shape by applying the inverse transformation matrix of the above-mentioned affine transformation matrix to the combined image. Subsequently, the converted combined image is used for an appropriate purpose. For example, in face verification, features for face verification are extracted from the converted combined image.
[0044] In the case where the third image frame is input to the image fusion device, the image fusion device detects a third feature point of the object shown in the third image frame from the third image frame in the same manner as described above with respect to the first image frame and the second image frame, and converts the third image frame based on the detected third feature point and the reference point. When it is assumed that the combined image is generated based on the three image frames, the image fusion device obtains a new combined image by combining the combined image obtained in operation 250 and the converted third image frame. The combined image of operation 250 and the converted third image frame are combined using the sum, average, weighted sum, or weighted average of pixel values at corresponding positions in the combined image and the third image frame. However, examples of combination are not limited thereto.
[0045] As described above, the image fusion device converts the image frames so that the feature points of the image frames are placed at fixed positions of the reference points, and fuses the converted image frames, thereby providing a combined image in which motion blur and noise are reduced and the features of the object are well represented. In the case of an image-based authentication system in which it is important to accurately extract the features of the object using the combined image, the accuracy of recognition is improved.
[0046] Figure 3 is a flowchart illustrating an example of an image fusion method.
[0047] Reference Figure 3 In operation 310, the image fusion device receives an nth image frame, where n is an integer greater than 0. In operation 320, the image fusion device measures the image quality of the received image frame. For example, the image fusion device measures the brightness or blur level of the image frame. The image quality of the image frame is used to determine whether to use the corresponding image frame to generate a combined image.
[0048] In operation 330, the image fusion device determines whether the measured image quality meets a preset condition. For example, the image fusion device determines whether the brightness of the image frame is less than a threshold value. In response to the image quality meeting the preset condition, the image fusion device performs operation 340. In response to the image quality not meeting the preset condition, the image fusion device performs operation 390 to select subsequent image frames. In this way, image frames taken in a bright environment are prevented from being used to generate a combined image, and it is determined whether image frames for image fusion are continuously received. This is because it is not necessary to perform image fusion on feature image frames that are taken in a bright environment and therefore have a brightness large enough to define an object. In one example, in the case where the image quality does not meet the preset condition, the currently executed image fusion process is terminated, and the subsequent image frame is designated as the first image frame of the image fusion process so that the image fusion process is restarted.
[0049] In operation 340, the image fusion device detects feature points of an object region and a target object from an image frame. In operation 350, the image fusion device generates a transformed image frame by performing image transformation (e.g., image deformation) on the image frame so that the feature points detected from the image frame are placed at fixed positions of corresponding reference points. In one example, the image fusion device extracts an area to be subjected to image transformation from the image frame based on the detected feature points; determines a transformation matrix for defining a transformation relationship between positions of feature points contained in the extracted area and positions of reference points; and generates a transformed image frame by applying the determined transformation matrix to the image frame or the corresponding extracted area.
[0050] In operation 360, the image fusion device obtains a combined image by image fusion. The image fusion device obtains the combined image by fusing the previously converted and stored image frame and the currently converted image frame. Here, the image fusion includes a process of fusing pixel values at corresponding positions between the converted image frames by sum, average, weighted sum, or weighted average of pixel values. However, examples of image fusion are not limited thereto.
[0051] In operation 370, the image fusion device performs specific image processing on the combined image for the purpose of using the combined image. For example, the image fusion device performs inverse image deformation on the combined image to restore the shape of the object shown in the combined image to its original shape. In this example, the inverse image deformation is performed to place the feature points of the object shown in the combined image at the positions of the corresponding feature points in the current image frame.
[0052] In operation 380, the image fusion apparatus determines whether there is a subsequent image frame. In the case that the subsequent image frame exists, the image fusion apparatus selects the subsequent image frame in operation 390. After the selection is made, the image fusion apparatus performs the processes of operations 320 to 380 again for the selected subsequent image frame.
[0053] Figure 4 is a flowchart illustrating an example of an image fusion method.
[0054] Reference Figure 4 , operations 410, 420, 430, 440, 450, 460, 470, 480, and 490 correspond to Figure 3 310, 340, 350, 320, 330, 360, 370, 380, 390 in FIG. Therefore, referring to Figure 3 The description provided applies to Figure 4 operation.
[0055] However, Figure 4 The example of the operation 440 of measuring the image quality and the operation 450 of determining whether the measured image quality satisfies the preset condition are performed after the operation 430 of performing the image conversion. Figure 3 is different from the example of . In addition, in operation 420, an object area is detected from the image frame. In operation 430, an image conversion (e.g., image deformation) is performed on the detected object area so that the feature point is placed at the position of the corresponding reference point. Then, in operation 440, the image quality (e.g., brightness) of the converted object area is measured. In operation 450, it is determined whether the image quality satisfies a preset condition. By considering the image quality of the object area as the area of interest, image fusion can be applied to image frames that are overall bright but contain dark object areas (e.g., image frames shot against the light).
[0056] Figure 5 is a flowchart illustrating an example of applying the image fusion method to face verification.
[0057] Face verification is a method of user authentication for determining whether a user attempting authentication is a valid user based on the user's facial information. For example, the image fusion method described herein is used to provide an image that more clearly displays facial features in a face authentication process, even though the image is taken in a low-light environment. In the case of image-based object recognition technology (e.g., face verification), how good the features of the object shown in the image are greatly affects the recognition accuracy. When the blur or noise level shown in the image is relatively large, it is relatively difficult to accurately extract the features of the object from the corresponding image. By performing image fusion processing based on the feature points as described above, the image fusion device generates an image in which the blur or noise is reduced while its brightness is improved.
[0058] Reference Figure 5 In operation 505, the face verification device receives the nth image frame of the query facial image, where n is an integer greater than 0. The query facial image is an image used for face verification. In operation 510, the face verification device detects a face region from the image frame, and detects facial key points from the detected face region. In order to detect the face region, a Haar-based cascaded adaboost classifier or a Viola-Jones detector is used. However, the example is not limited to this. In one example, an image normalization process (e.g., image scaling or image cropping) is performed on the face region of the image frame.
[0059] In operation 515, the face verification device performs image transformation on the face region so that the face region is transformed. For example, the face verification device performs image deformation on the face region so that the facial key points detected from the face region are placed at the positions of the reference points in the predefined face region.
[0060] In operation 520, the image verification device measures the image quality of the converted face area. In operation 525, the face verification device determines whether the image quality satisfies a preset condition. For example, the face verification device measures the brightness of the converted face area and determines whether the measured brightness is less than a threshold value.
[0061] In operation 530, in response to the image quality satisfying the preset condition, the face verification device combines the face region converted in the previous image frame and the face region converted in operation 515 through image fusion to obtain a combined image. The process includes a process of fusing pixel values at corresponding positions in the converted face region. In one example, in addition to the two image frames, the face verification device also obtains the combined image by combining the converted face regions of at least three image frames.
[0062] In operation 535, the face verification device performs image processing on the combined image so that the obtained combined image has a shape more suitable for face verification. For example, the face verification device performs a process of restoring the shape of the face to its original shape by performing inverse image deformation, gamma correction for clearly expressing facial features shown in the combined image, or image processing for reducing color difference on the combined image.
[0063] In operation 550, the face verification device extracts features from the processed combined image. For example, the face verification device uses a neural network-based feature extractor to extract features or feature vectors. In response to determining in operation 525 that the image quality of the face region included in the current image frame does not meet a preset condition, for example, in response to the brightness of the detected face region being greater than or equal to a threshold, the face verification device extracts features from the current image frame in operation 550 without performing operations 530 and 535. In this example, the image fusion process is initialized so that the previously stored results of image conversion of the image frame and the combined image are removed from a storage (e.g., memory).
[0064] In operation 555, the image verification device determines whether the face verification is successful based on the extracted features. The face verification device calculates a matching score based on the similarity between the extracted features and the pre-registered features, and determines that the face verification is successful in response to the calculated matching score value satisfying a preset condition (e.g., in response to the matching score being greater than a threshold). For example, the matching score is determined based on the similarity or distance between the extracted feature vector and the pre-registered feature vector.
[0065] In operation 560, the face verification device determines whether the face verification is successful. In response to the face verification being unsuccessful (e.g., failed), the face verification device determines whether there is a subsequent image frame in operation 540. In response to determining that there is no subsequent image frame, the face verification device determines that the face verification has failed. In operation 545, in response to determining that there is a subsequent image frame, the face verification device selects a subsequent image frame and repeats the above process for the subsequent image frame.
[0066] Through the above processing, although face authentication is performed based on a face image captured in a low-brightness environment, by performing image fusion processing based on feature points, face authentication can be performed more accurately even in a low-brightness environment.
[0067] Figure 6 An example of an image fusion method is shown.
[0068] Reference Figure 6, an example of an image fusion method (specifically, an example of performing image fusion on a first image frame 612 and a second image frame 614 containing a license plate) is shown. In one example, the first image frame 612 and the second image frame 614 are image frames that are continuous in time or image frames that are adjacent in time but not immediately connected. For example, the first image frame 612 and the second image frame 614 are the nth image frame and the (n+1)th image frame, respectively, or the nth image frame and the (n+m)th image frame, respectively. Here, n is an integer greater than 0, and m is a natural number greater than or equal to "2".
[0069] When the first image frame 612 is input, the brightness of the first image frame 612 is measured. In response to determining that the first image frame 612 is acquired in a dark environment because the measured brightness is less than a threshold value, the image fusion device detects feature points 622 from the first image frame 612. For example, four vertices of the license plate are detected as feature points 622. The area of the license plate is detected from the first image frame 612 based on the detected feature points 622. Then, the image fusion device generates a transformed first image frame 632 by performing image deformation so that the feature points 622 are placed at the positions of predefined reference points.
[0070] A similar process is also performed for the second image frame 614. When the second image frame 614 is input, the brightness of the second image frame 614 is measured. In response to the measured brightness being less than a threshold value, the image fusion device detects a feature point 624 from the second image frame 614. The area of the license plate is detected from the second image frame 614 based on the detected feature point 624. Then, the image fusion device generates a transformed second image frame 634 by performing image deformation so that the feature point 624 is placed at the position of a predefined reference point.
[0071] In one example, the brightness of the image frame is measured after performing image deformation. In this example, the measured brightness of the image frame should be less than a preset threshold value to continuously perform image fusion processing based on the corresponding image frames. If the measured brightness is greater than or equal to the threshold value, the image fusion processing is terminated or initialized.
[0072] The image fusion device generates a combined image 640 by combining the converted first image frame 632 and the converted image frame 634. For example, the image fusion device generates the combined image 640 by fusing the pixel values of the converted first image frame 632 and the pixel values of the converted second image frame 634. Then, the image fusion device generates a restored combined image 650 by restoring the shape of the license plate shown in the combined image 640 to its original shape shown in the image frames. For example, the image fusion device restores the shape of the license plate by applying an inverse transformation matrix of the transformation matrix used for the above-mentioned image deformation to the combined image 640.
[0073] Figure 7 An example of an image fusion method is shown.
[0074] Reference Figure 7 , an example of an image fusion method (specifically, an example of performing image fusion on a first image frame 712 and a second image frame 714 containing a face area) is shown. In one example, the first image frame 712 and the second image frame 714 are image frames that are continuous in time or image frames that are adjacent in time but not immediately connected. For example, the first image frame 712 and the second image frame 714 are the nth image frame and the (n+1)th image frame, respectively, or the nth image frame and the (n+m)th image frame, respectively. Here, n is an integer greater than 0, and m is a natural number greater than or equal to "2".
[0075] When the first image frame 712 is input, the brightness of the first image frame 712 is measured. In response to the measured brightness being less than a threshold value, the image fusion device detects facial key points 722 as feature points from the first image frame 712. For example, feature points of major facial parts (e.g., eyebrows, eyes, nose, and mouth) are detected. A face region 732 is detected from the first image frame 712 based on the detected key points 722. The detected face region 732 is extracted as a face image 742 corresponding to the face region 732 by image cropping. Then, the image fusion device generates a converted face image 762 by performing image deformation on the face image 742 so that the feature points shown in the face image 742 are placed at the positions of the predefined reference points 750. For example, image deformation is performed so that the feature points of the eyebrows, nose, and mouth shown in the face image 742 are placed at the positions of the reference points 750 corresponding to the eyebrows, nose, and eyes, respectively.
[0076] A similar process is also performed for the second image frame 714. When the second image frame 714 is input, the brightness of the second image frame 714 is measured. In response to the measured brightness being less than a threshold value, the image fusion device detects facial key points 724 from the second image frame 714. A face region 734 is detected from the second image frame 714 based on the detected key points 724. The detected face region 734 is extracted as a face image 744 corresponding to the face region 734 through image cropping. Then, the image fusion device generates a converted face image 764 by performing image deformation on the face image 744 so that the feature points shown in the face image 744 are placed at the position of the predefined reference point 750.
[0077] In one example, the brightness of the image frame is measured after performing image deformation. In this example, the measured brightness of the image frame should be less than a preset threshold value to continuously perform image fusion processing based on the corresponding image frames. If the measured brightness is greater than or equal to the threshold value, the image fusion processing is terminated or initialized.
[0078] The image fusion device generates a combined image 770 by combining the converted image frame 762 and the converted image frame 764. For example, the image fusion device generates the combined image 770 using the sum, average, weighted sum, or weighted average of the pixel values of the converted face image 762 and the pixel values of the converted face image 764. Then, the image fusion device generates a restored combined image 775 by restoring the shape of the face shown in the combined image 770 to its original shape shown in the image frame. The restored combined image 775 is applied to the second image frame 714. Through the above processing, a result image in which the brightness of the face area is improved and the motion blur and noise are reduced is obtained. Therefore, in the case of performing face verification using the corresponding result image, facial features are more accurately extracted, thereby improving the accuracy of face verification.
[0079] exist Figure 6 and Figure 7 In the example of , image fusion is performed based on two image frames. However, the example is not limited thereto. Image fusion may also be performed for at least three image frames.
[0080] Figure 8 An example of the configuration of an image fusion device is shown.
[0081] Reference Figure 8 , the image fusion device 800 receives a series of image frames. The image fusion device 800 detects feature points of the object from the image frames, and performs image fusion between the image frames based on the detected feature points. In this example, the image fusion device 800 determines whether to perform image fusion based on the image quality (e.g., brightness) of the image frames. The image fusion device 800 performs at least one operation of the image fusion method described or illustrated herein, and provides the result of face verification to the user.
[0082] The image fusion device 800 includes at least one processor 810 and a memory 820. The processor 810 executes instructions to perform a reference Figures 1 to 7 At least one operation described. For example, the processor 810 detects a first feature point of an object shown in the first image frame from a first image frame, and transforms the first image frame based on the detected first feature point and a predefined reference point. For example, the processor 810 transforms the first image frame by performing image deformation so that the first feature point is placed at the position of the reference point. In another example, the processor 810 detects a first object area from the first image frame based on the detected first feature point, and performs image deformation on the first object area based on the first feature point and the reference point contained in the detected first object area.
[0083] In addition, the processor 810 detects a second feature point of an object shown in the second image frame from the second image frame, and transforms the second image frame based on the detected second feature point and the reference point. For example, the processor 810 transforms the second image frame by performing image deformation so that the second feature point is placed at the position of the reference point. In another example, the processor 810 detects a second object area from the second image frame based on the detected second feature point, and performs image deformation on the second object area based on the second feature point and the reference point contained in the detected second object area.
[0084] The processor 810 generates a combined image by combining the converted first image frame and the converted second image frame. The processor 810 generates the combined image by combining the pixel value of each pixel of the converted first image frame and the pixel value of each pixel of the converted second image frame, for example, using the sum, average, weighted sum, or weighted average of the pixel value of each pixel of the converted first image frame and the pixel value of each pixel of the converted second image frame. The processor 810 converts the combined image based on the correspondence between the second feature point of the second image frame and the reference point. This includes a process of restoring the shape of the object converted during the image fusion process to its original shape based on the position of the feature point shown in the second image frame.
[0085] The memory 820 is connected to the processor 810 and stores instructions executable by the processor 810 and data to be calculated by the processor 810 or data processed by the processor 810. The memory 820 includes a non-transitory computer-readable medium, such as a high-speed random access memory (RAM) and / or a non-volatile computer-readable storage medium.
[0086] Fig. 9 An example of a configuration of a computing device is shown.
[0087] Reference Fig. 9 , the computing device 900 is a device that executes an application using the above-mentioned image fusion method. In one example, the computing device 900 and Figure 5 For example, computing device 900 is an image processing device, a smart phone, a wearable device, a tablet computer, a netbook computer, a laptop computer, a desktop computer, a personal digital assistant (PDA), a set-top box, a home appliance, a biometric door lock, a security device, or a vehicle starting device. Computing device 900 includes Figure 8 The image fusion device 800 has the same functions as the others.
[0088] Reference Fig. 9The computing device 900 includes a processor 910, a storage device 920, a camera 930, an input device 940, an output device 950, and a network interface 960. The processor 910, the storage device 920, the camera 930, the input device 940, the output device 950, and the network interface 960 communicate with each other via a communication bus 970.
[0089] The processor 910 executes functions and instructions to be executed by the computing device 900. For example, the processor 910 processes instructions stored in the storage device 920. The processor 910 executes the instructions with reference to Figures 1 to 8 Describes at least one operation.
[0090] The storage device 920 stores information or data required for the execution of the processor 910. The storage device 920 includes a computer-readable storage medium or a computer-readable storage device. The storage device 920 stores instructions to be executed by the processor 910 and stores relevant information when the computing device 900 executes an application or software.
[0091] The camera 930 captures an image including a plurality of image frames. For example, the camera 930 captures a user attempting facial verification and acquires a query facial image including a plurality of image frames.
[0092] The input device 940 receives input from the user through tactile, video, audio or touch input. For example, the input device 940 includes: a keyboard, a mouse, a touch screen, a microphone, or other predetermined devices that detect input from the user and send the detected input to the computing device 900.
[0093] The output device 950 provides the output of the computing device 900 to the user through visual, auditory or tactile channels. For example, the output device 950 includes: a display, a touch screen, a speaker, a vibration generator or other predetermined devices that provide output to the user. The network interface 960 communicates with external devices through a wired or wireless network.
[0094] Image fusion devices 100 and 800, computing device 900 and other devices, units, modules, apparatuses and the like Figure 1 and Figure 8 to Figure 9The other components described are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application may appropriately include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In another example, one or more hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (e.g., logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programming logic arrays, microprocessors, or any other devices or combinations of devices configured to respond and execute instructions in a limited manner to obtain desired results). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer may be executable instructions or software (e.g., an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. Hardware components can also access, operate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor or a controller, and one or more other hardware components can be implemented by one or more other processors, or other processors and other controllers. One or more processors, or a processor and a controller can implement a single hardware component, or two or more hardware components. Hardware components can have any one or more different processing configurations, wherein the examples of one or more different processing configurations include: single processor, independent processor, parallel processor, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, multiple instruction multiple data (MIMD) multiprocessing.
[0095] Figures 2 to 7The method for performing the operations described in the present application is shown to be performed by computing hardware (e.g., one or more processors or computers), wherein the computing hardware is implemented to execute instructions or software as described above to perform the operations described in the present application performed by the method. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation, or two or more operations.
[0096] The instructions or software for controlling a processor or computer to implement a hardware component and perform the method described above are written as a computer program, a code snippet, an instruction, or any combination thereof, to individually or collectively instruct or configure a processor or computer to operate as a machine or a special-purpose computer to perform the operations performed by the hardware component and the method described above. In one example, the instructions or software include machine code (e.g., machine code generated by a compiler) directly executed by a processor or computer. In another example, the instructions or software include high-level code executed by a processor or computer using an interpreter. Ordinary programmers in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware component and the method described above.
[0097] Instructions or software for controlling a processor or computer-implemented hardware component and performing the method described above, and any associated data, data files, and data structures are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (e.g., multimedia card or micro card (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store instructions or software and any associated data, data files and data structures in a non-temporary manner, and provide instructions or software and any associated data, data files and data structures to a processor or computer so that the processor and computer can execute the instructions.
[0098] Although the present disclosure includes specific examples, it will be clear to those of ordinary skill in the art that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein will be considered to be descriptive only and not for the purpose of limitation. The description of the features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, suitable results can be achieved. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as included in the present disclosure.
Claims
1. An image fusion method, include: measuring the brightness of the first image frame; In response to the measured brightness of the first image frame being less than a threshold, detecting, from the first image frame, a first feature point of an object shown in the first image frame; transforming the first image frame based on the detected first feature point and the predefined reference point; measuring the brightness of the second image frame; In response to the measured brightness of the second image frame being less than a threshold, detecting, from the second image frame, a second feature point of the object shown in the second image frame; converting the second image frame based on the detected second feature points and the reference point; obtaining a combined image showing the object by combining the converted first image frame and the converted second image frame, The first image frame and the second image frame are image frames that are continuous in time or image frames that are adjacent in time but not closely connected. The image fusion method further includes: performing inverse image deformation on the combined image to place a feature point of the object shown in the combined image at a position of a second feature point in the second image frame to restore the shape of the object shown in the combined image to the original shape of the object, The reference point has a pre-fixed position according to the type of the object.
2. The image fusion method according to claim 1, in, The step of converting the first image frame includes: converting the first image frame so that the first feature point is placed at the position of the reference point, The step of converting the second image frame includes converting the second image frame so that the second feature point is placed at the position of the reference point.
3. The image fusion method according to claim 1, in, The step of converting the first image frame comprises: detecting a first object region from the first image frame based on the detected first feature points; transforming the first object region by image deformation based on first feature points and reference points contained in the detected first object region, The step of converting the second image frame comprises: detecting a second object region from the second image frame based on the detected second feature points; The second object region is converted through image deformation based on second feature points and reference points included in the detected second object region.
4. The image fusion method according to claim 3, in, The step of obtaining includes combining the converted first object region and the converted second object region.
5. The image fusion method according to claim 1, in, The step of obtaining includes generating a combined image by combining first pixel values of the converted first image frame and second pixel values of the converted second image frame.
6. The image fusion method according to claim 5, in, The combination of the first pixel value and the second pixel value includes one of a sum, an average, a weighted sum, and a weighted average of the first pixel value and the second pixel value at corresponding positions in the converted first image frame and the converted second image frame.
7. The image fusion method according to claim 1, further comprising: include: measuring an image quality of a converted first image frame; In response to the measured image quality satisfying a preset condition, it is determined to obtain a combined image based on the converted first image frame.
8. The image fusion method according to claim 7, in, The measuring step includes: measuring the brightness of the first image frame, The determining step includes: in response to the measured brightness of the first image frame being less than a preset threshold, determining to obtain a combined image based on the converted first image frame.
9. The image fusion method according to claim 1, further comprising: include: measuring an image quality of the converted second image frame; In response to the measured image quality satisfying a preset condition, it is determined to obtain a combined image based on the converted second image frame.
10. The image fusion method according to claim 1, further comprising: include: detecting, from the third image frame, third feature points of the object shown in the third image frame; converting a third image frame based on the detected third feature points and the reference point; The obtained combined image and the converted third image frame are combined.
11. The image fusion method according to claim 1, in, The step of detecting the first feature point comprises: detecting facial key points from the first image frame, The step of detecting the second feature points includes: detecting facial key points from the second image frame.
12. The image fusion method according to claim 1, further comprising: include: The combined image is converted based on the correspondence between the second feature point and the reference point.
13. The image fusion method according to claim 12, in, Features for face verification are extracted from the transformed combined image. 14 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, causes the processor to execute the image fusion method of claim 1 .
15. An image fusion device, include: processor, The processor is configured as follows: measuring the brightness of the first image frame; In response to the measured brightness of the first image frame being less than a threshold, detecting, from the first image frame, a first feature point of an object shown in the first image frame; transforming the first image frame based on the detected first feature point and the predefined reference point; measuring the brightness of the second image frame; In response to the measured brightness of the second image frame being less than a threshold, detecting, from the second image frame, a second feature point of the object shown in the second image frame; converting the second image frame based on the detected second feature points and the reference point; generating a combined image showing the object by combining the converted first image frame and the converted second image frame, The first image frame and the second image frame are image frames that are continuous in time or image frames that are adjacent in time but not closely connected. The processor is further configured to: perform inverse image deformation on the combined image to place a feature point of the object shown in the combined image at a position of a second feature point in the second image frame, so as to restore the shape of the object shown in the combined image to the original shape of the object, The reference point has a pre-fixed position according to the type of the object.
16. The image fusion device according to claim 15, in, The processor is further configured to: generating a transformed first image frame by transforming the first image frame so that the first feature point is placed at the position of the reference point, A converted second image frame is generated by converting the second image frame so that the second feature point is placed at the position of the reference point.
17. The image fusion device according to claim 15, in, The processor is further configured to: detecting a first object region from the first image frame based on the detected first feature points, and transforming the first object region through image deformation based on the first feature points and reference points included in the detected first object region; A second object region is detected from the second image frame based on the detected second feature points, and the second object region is converted through image deformation based on the second feature points and the reference point included in the detected second object region.
18. The image fusion device according to claim 15, in, The processor is further configured to generate a combined image by combining the first pixel values of the converted first image frame and the second pixel values of the converted second image frame.
19. The image fusion device according to claim 18, in, The combination of the first pixel value and the second pixel value includes one of a sum, an average, a weighted sum, and a weighted average of the first pixel value and the second pixel value at corresponding positions in the converted first image frame and the converted second image frame.
20. The image fusion device according to claim 15, in, The processor is further configured to transform the combined image based on the correspondence between the second feature point and the reference point.
Citation Information
Patent Citations
Connection structure of circuit substrate
KR1020180016451A
Image capture apparatus and method
CN101547316A
Infant image composition method and device
CN103295211A
Electronic aperture shooting method and terminal
CN105915796A
Image noise reduction
CN105976327A
