Method and apparatus for correcting brightness in multi-view image
Patent Information
- Application Number
- KR1020250028247
- Authority / Receiving Office
- KR · KR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-03-05
- Publication Date
- 2026-09-14
Smart Images
Figure PAT00005_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a method and apparatus for correcting the brightness of multi-view images, and more specifically, to a method and apparatus for correcting the brightness of multi-view images so that the brightness of the multi-view images is corrected to be similar to one another. Background Technology
[0002] There are various mathematically modeled methods for correcting the brightness of a single image, such as histogram equalization and gamma correction. However, when applying such techniques to each image in multi-view images for 3D spatial reconstruction, where camera characteristics and lighting environments differ, there is a disadvantage in that consistent results cannot be obtained.
[0003] Meanwhile, in the field of color transfer, which involves transferring the color distribution of a reference image to a target image from two or more images, there exist various methods utilizing deep learning, as well as existing computational methods that utilize the mean and variance of the global color distribution of the images or local correspondence relationships. However, similar to the above, consistent brightness and color correction results for multi-view images cannot be guaranteed, and there is a disadvantage in that correction is performed without considering accurate correspondence relationships, particularly for objects of interest where consistency in brightness and lighting is important, resulting in inconsistencies in surface continuity in the 3D reconstruction results.
[0004] Korean registered patent 10-0809349 discloses a technology that divides a single image into multiple divided regions, calculates the brightness distribution of each divided region to determine the main region of the image, and corrects the brightness of the main region to a target brightness.
[0005] Korean registered patent 10-0925629 discloses a technology that sets multiple intervals in the YCbCr channel in a single image, calculates the cumulative value of each interval to determine an effective interval for automatic white balance, calculates the average YCbCr value for correction, converts it to an RGB channel, and reflects the white balance gain.
[0006] Korean Registered Patent No. 10-0668073 discloses a technology for compensating brightness uniformity for each camera by taking images of four or more cameras so that the entire image fills the same plane, converting each captured image into a YCbCr space to extract a Y channel, calculating a reference brightness value from the Y channel for each camera, calculating a brightness difference map using the reference brightness value, and summing the Y channel and the brightness difference map. The problem to be solved
[0007] The problem that the present invention aims to solve is to provide a method and apparatus for correcting the brightness of a multi-view image that can consistently correct the brightness in a multi-view image.
[0008] Another problem that the present invention aims to solve is to provide a method and apparatus for correcting the brightness of a multi-view image that can resolve inconsistencies in the surface permanence of a 3D reconstruction result.
[0009] Another problem that the present invention aims to solve is to provide a method and apparatus for correcting the brightness of a multi-view image that can consistently correct the brightness in the multi-view image to match the brightness of the captured scene. means of solving the problem
[0010] A method for correcting the brightness of a multi-view image according to the present invention may include the steps of: calculating tie points in a plurality of multi-view images; storing the color value of a pixel corresponding to the tie point in the multi-view image used for calculating the tie point in association with the tie point; converting the stored color value of the pixel into a YCbCr channel; selecting a reference image among the plurality of multi-view images based on at least one of a luminance (Y) value among the converted color values and a pre-set rule; calculating a brightness correction graph between the selected reference image and another multi-view image based on the Y value; and correcting the brightness of the other multi-view image using the calculated brightness correction graph. Here, the color value of the pixel corresponding to the tie point may be stored within the data structure of the tie point. Additionally, the pre-set rule may be set by a user.
[0011] The step of storing in association with the above tie point may include: a step of extracting feature points from the plurality of multi-view images and aligning the extracted feature points to calculate the correspondence relationship of the feature points; a step of estimating camera geometric information of the plurality of multi-view images based on the calculated correspondence relationship; and a step of calculating the tie point based on the estimated camera geometric information and storing the color value of the pixel corresponding to the tie point in the multi-view image used when calculating the tie point in association with the tie point.
[0012] The step of selecting the reference image may include classifying the converted color values into each of the plurality of multi-view images to generate a set of color values for each multi-view image, calculating the median of each color value in the generated set of color values, and selecting the reference image based on at least one of the calculated median and a pre-set rule.
[0013] The step of calculating the brightness correction graph may include the step of calculating the brightness correction graph based on the correspondence relationship between the Y value of the reference image and the Y value of the other multi-view image with respect to each of the tie points.
[0014] The step of calculating the brightness correction graph above includes the step of calculating the brightness correction graph using an error measurement method or a monotonically increasing third-order polynomial curve fitting technique, and the error measurement method may include a Root Mean Square Error (RMSE) technique.
[0015] The step of correcting the brightness of the multi-view image may include the step of converting the color values of the corrected multi-view image into RGB channels and storing them.
[0016] The above feature points can be extracted from each multi-view image using the Scale Invariant Feature Transform (SIFT) technique.
[0017] The above camera geometric information can be estimated using one of the PnP technique, Bundle adjustment technique, Reduced Camera System technique, Conjugate Gradient technique, or SfM (Structure from Motion) technique.
[0018] An apparatus for correcting the brightness of a multi-view image according to the present invention may include a storage unit for storing a plurality of multi-view images, and a control unit that calculates a tie point in the plurality of multi-view images, stores a color value of a pixel corresponding to the tie point in the multi-view image used when calculating the tie point in association with the tie point in the storage unit, converts the stored color value of the pixel into a YCbCr channel, selects a reference image among the plurality of multi-view images based on at least one of a luminance (Y) value among the converted color values and a preset rule, calculates a brightness correction graph between the selected reference image and another multi-view image using the Y value, and corrects the brightness of the other multi-view image using the calculated brightness correction graph. Here, the color value of the pixel corresponding to the tie point may be stored within the data structure of the tie point. Additionally, the preset rule may be set by a user.
[0019] The control unit extracts feature points from the plurality of multi-view images, aligns the extracted feature points to calculate the correspondence of the feature points, estimates camera geometric information of the plurality of multi-view images based on the calculated correspondence, calculates the tie point based on the estimated camera geometric information, and stores the color value of the pixel corresponding to the tie point in the multi-view image used when calculating the tie point in association with the tie point in the storage unit.
[0020] The control unit classifies the converted color values into each of the plurality of multi-view images to generate a set of color values for each multi-view image, calculates the median of each color value in the generated set of color values, and can select the reference image based on at least one of the calculated median and a pre-set rule.
[0021] The control unit above can calculate the brightness correction graph based on the correspondence relationship between the Y value of the reference image and the Y value of the other multi-view image with respect to each of the tie points.
[0022] The control unit calculates the brightness correction graph using an error measurement method or a monotonically increasing third-order polynomial curve fitting technique, and the error measurement method may include a Root Mean Square Error (RMSE) technique.
[0023] The control unit above can convert the color values of the corrected multi-view image into RGB channels and store them in the storage unit.
[0024] The above feature points can correct the brightness of the multi-view images extracted from each multi-view image using the Scale Invariant Feature Transform (SIFT) technique.
[0025] The above camera geometric information can be estimated using one of the PnP technique, Bundle adjustment technique, Reduced Camera System technique, Conjugate Gradient technique, or SfM (Structure from Motion).
[0026] The present invention may include a computer program stored on a medium readable by a computing device to execute a method for correcting the brightness of a multi-view image according to the present invention.
[0027] The present invention may include a non-transient computer-readable storage medium storing instructions, wherein the instructions, when executed by a processor, cause the processor to perform an operation associated with a method for correcting the brightness of a multi-view image according to the present invention or a method for correcting the brightness of a multi-view image according to the present invention.
[0028] The present invention may include a computer program product comprising an executable command for executing a method for correcting the brightness of a multi-view image according to the present invention. Effects of the invention
[0029] According to the method and apparatus for correcting the brightness of a multi-view image according to the present invention, the brightness correction graph calculated according to the configuration of the present invention is calculated through an accurately predicted correspondence for each image, so uniform brightness correction performance can be guaranteed for all images. In addition, the present invention has the advantage of being able to correct the brightness of a multi-view image with only simple calculations in an intermediate stage during a 3D space reconstruction process using a multi-view image. Brief explanation of the drawing
[0030] Figure 1 is a diagram illustrating a method for capturing multi-view images. Figure 2 is a diagram illustrating another method of capturing multi-view images. FIG. 3 is a block diagram illustrating the configuration of a device for correcting the brightness of a multi-view image according to an embodiment of the present invention. FIG. 4 is a block diagram illustrating the configuration of a tie point calculation unit according to an embodiment of the present invention. FIG. 5 is a flowchart illustrating the process of performing a method for correcting the brightness of a multi-view image according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating the process of performing a tie point calculation method according to an embodiment of the present invention. FIG. 7 is a block diagram showing the configuration of a device for correcting the brightness of a multi-view image according to another embodiment of the present invention. Specific details for implementing the invention
[0031] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.
[0032] The detailed description of the exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description below is not intended to be taken in a limiting sense, and the scope of the exemplary embodiments is limited only by the appended claims, with all equivalents to those claimed therein, provided appropriately described.
[0033] Similar reference numerals in the drawings refer to the same or similar functions across multiple aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation. Additionally, the term "and / or" may include a combination of multiple related described items or any of the multiple related described items. The terms "part," "unit," and "module" as used in the present invention may include one or more components, and may include software components and / or hardware components.
[0034] In the present invention, terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.
[0035] When it is stated that a component is "connected" or "connected" to another component, it should be understood that the two components may be directly connected or connected to each other, or that another component may exist between the two components. On the other hand, when it is stated that a component is "directly connected" or "directly connected" to another component, it should be understood that no other component exists between the two components.
[0036] The components shown in the embodiments are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the present invention as long as they do not depart from the essence of the invention.
[0037] The terms used in the embodiments are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the embodiments, terms such as "comprising" or "having" are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, the description in the embodiments that a specific configuration "comprising" does not exclude configurations other than that configuration, and means that additional configurations may also be included within the scope of the practice or technical concept of the invention.
[0038] In the embodiments, the term “at least one” may mean one of a number of 1 or more, such as 1, 2, 3, and 4. In the embodiments, the term “a plurality of” may mean one of a number of 2 or more, such as 2, 3, and 4.
[0039] Some components of the embodiments may not be essential components performing an essential function in the present invention, but merely optional components for enhancing performance. The embodiments may be implemented by including only the components essential for embodying the essence of the embodiments, excluding the components used solely for performance enhancement. A structure including only the essential components, excluding the optional components used solely for performance enhancement, is also included within the scope of the embodiments.
[0040] In the following, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the embodiments. In describing the embodiments, detailed descriptions of related known components or functions are omitted if it is determined that such detailed descriptions may obscure the gist of this specification. Additionally, the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.
[0041] In the following, "image" may refer to a picture constituting a video, or it may refer to the video itself. For example, "synthesis and / or creation of an image" may mean "synthesis and / or creation of a video," and may also mean "synthesis and / or creation of one of the images constituting a video."
[0042] In the following, the terms "video" and "motion picture(s)" may be used interchangeably with the same meaning.
[0043] In the following, the terms "image," "picture," and "frame" may be used interchangeably with the same meaning.
[0044] Figure 1 is a diagram illustrating a method for capturing multi-view images.
[0045] Referring to FIG. 1, when taking a multi-view image of a three-dimensional object (991), the camera (981) can be moved 360 degrees so as to face the three-dimensional object (991), and a multi-view image can be taken of the three-dimensional object (991) at each set shooting point. Alternatively, a camera can be installed at each set shooting point, and the camera installed at each shooting point can take a multi-view image of the three-dimensional object (991).
[0046] Figure 2 is a diagram illustrating another method of capturing multi-view images.
[0047] Referring to FIG. 2, when the three-dimensional object (992) has an area of some width, a drone (982) equipped with a camera can fly in a zigzag pattern and capture a multi-view image of the three-dimensional object (992).
[0048] Multi-view images can be captured with a single camera or multiple cameras, as shown in FIGS. 1 and 2. When captured with a single camera, there may be an inherent difference in brightness between the captured multi-view images depending on the time and location of the shot. Additionally, when captured with multiple cameras, there may be an inherent difference in brightness between the cameras and between the captured multi-view images depending on the location of the shot.
[0049] The present invention can correct brightness differences between multi-view images that may inherently exist during the capture of multi-view images, and can resolve inconsistencies in surface continuity caused by brightness differences between multi-view images in a 3D reconstruction result.
[0050] FIG. 3 is a block diagram illustrating the configuration of a device for correcting the brightness of a multi-view image according to an embodiment of the present invention.
[0051] Referring to FIG. 3, the device (300) for correcting the brightness of a multi-view image according to the present invention may include a tie point calculation unit (310), a channel conversion unit (320), a reference image selection unit (330), a brightness correction graph calculation unit (340), and a brightness correction unit (350). The device (300) for correcting the brightness of a multi-view image may be implemented in a computer system including a recording medium that can be read by a computer.
[0052] The tie point calculation unit (310) calculates tie points from a plurality of multi-view images and can store the color value of a pixel corresponding to the tie point in the multi-view images used for tie point calculation in association with the tie point. Here, the plurality of multi-view images may be images stored in advance or images input from at least one camera. The plurality of multi-view images may be images received from a camera, an electronic device, or a server. Additionally, the plurality of multi-view images may be images captured in the manner described in FIG. 1 or FIG. 2.
[0053] FIG. 4 is a block diagram illustrating the configuration of a tie point calculation unit according to an embodiment of the present invention.
[0054] Referring to FIG. 4, the tie point calculation unit (310) may include a feature point extraction unit (410), a camera geometric information estimation unit (420), and a color value processing unit (430).
[0055] The feature point extraction unit (410) can extract feature points from multiple multi-view images and calculate the correspondence of feature points by aligning the extracted feature points. The feature point extraction unit (410) can extract feature points from each multi-view image using a Scale Invariant Feature Transform (SIFT) technique. Additionally, the feature point extraction unit (410) can calculate the correspondence of feature points by aligning the extracted feature points using the SIFT technique. The technique for calculating the correspondence of feature points from multiple multi-view images can utilize techniques known in the art in addition to the SIFT technique, and a detailed description is omitted here.
[0056] The camera geometric information estimation unit (420) can estimate camera geometric information of multiple multi-view images based on the correspondence of feature points calculated by the feature point extraction unit (410). The camera geometric information can be estimated using one of the PnP technique, Bundle adjustment technique, Reduced Camera System technique, Conjugate Gradient technique, or SfM (Structure from Motion) technique. Additionally, the technique for estimating the camera geometric information may utilize other techniques known in the art, and a detailed description is omitted here.
[0057] The color value processing unit (430) calculates a tie point based on the camera geometric information estimated by the camera geometric information estimation unit (420), and can store the color value of a pixel corresponding to the tie point in the multi-view image used when calculating the tie point in association with the tie point. The color value processing unit (430) can store the color value of the pixel corresponding to the tie point within the data structure of the corresponding tie point.
[0058] The channel conversion unit (320) can convert the color value of the pixel stored by the tie point calculation unit (310) in association with the tie point into a YCbCr channel.
[0059] The reference image selection unit (330) can select a reference image from among a plurality of multi-view images based on at least one of the luminance (Y) value among the color values converted by the channel conversion unit (320) and a preset rule.
[0060] The reference image selection unit (330) can classify the color values converted by the channel conversion unit (320) into each of a plurality of multi-view images to generate a set of color values for each multi-view image. Here, each set of color values may include the converted color values of the related multi-view image.
[0061] The reference image selection unit (330) can calculate the median of each color value in the generated set of color values. In some embodiments, the reference image selection unit (330) can select a reference image based on the calculated median.
[0062] In some embodiments, the reference image selection unit (330) may select a reference image based on at least one of a preset rule. Here, the preset rule may be set by a user, and the preset rule may include a rule for selecting a reference image using each color value from a set of generated color values.
[0063] The brightness correction graph calculation unit (340) can calculate a brightness correction graph between a brightness reference image selected by the reference image selection unit (340) and another multi-view image based on the Y value. Here, the brightness correction graph calculation unit (340) can calculate the brightness correction graph based on the correspondence relationship between the Y value of the brightness reference image and the Y value of another multi-view image for each tie point. The Y value of the brightness reference image may be a Y value included in the set of color values of the reference image, and the Y value of the other view image may be a Y value included in the set of color values of the other view image.
[0064] The brightness correction graph calculation unit (340) can calculate the brightness correction graph using an error measurement method or a monotonically increasing third-order polynomial curve fitting technique. The error measurement method may include the Root Mean Square Error (RMSE) technique and other techniques known in the art, and a detailed description is omitted here.
[0065] The brightness correction unit (350) can correct the brightness of other multi-view images using the brightness correction graph calculated by the brightness correction graph calculation unit (340). The brightness correction unit (350) can correct the brightness of the remaining multi-view images, excluding the reference image among the multiple multi-view images, based on the brightness correction graph calculated by the brightness correction graph calculation unit (340).
[0066] The brightness correction unit (350) can convert the color values of the corrected multi-view image into RGB channels and store them.
[0067] FIG. 5 is a flowchart illustrating the process of performing a method for correcting the brightness of a multi-view image according to an embodiment of the present invention.
[0068] Referring to FIG. 5, the tie point calculation unit (310) calculates tie points from a plurality of multi-view images and stores the color value of a pixel corresponding to the tie point in the multi-view images used for tie point calculation in association with the tie point (S100). Here, the plurality of multi-view images may be images stored in advance or images input from at least one camera. The plurality of multi-view images may be images received from a camera, an electronic device, or a server. Additionally, the plurality of multi-view images may be images captured in the manner described in FIG. 1 or FIG. 2.
[0069] Step S100 may include the process of performing the tie point calculation method according to the present invention.
[0070] FIG. 6 is a flowchart illustrating the process of performing a tie point calculation method according to an embodiment of the present invention.
[0071] Referring to FIG. 6, the feature point extraction unit (410) extracts feature points from a plurality of multi-view images (S200). In step S200, the feature point extraction unit (410) can extract feature points from each multi-view image using the SIFT technique. Techniques known in the art other than SIFT can be used as methods for extracting feature points, and a detailed description is omitted here.
[0072] In the case where there are n multiple multi-view images, in step S200, the feature point extraction unit (410) first performs for all n multi-view images each multi-view image I i Feature points can be extracted from (i is a value between 1 and n).
[0073] The feature point extraction unit (410) aligns the extracted feature points to calculate the correspondence of the feature points (S210). In step S200, the feature point extraction unit (410) can use the SIFT technique to align the feature points of each multi-view image and calculate the correspondence of the feature points. Techniques known in the art other than SIFT can be used as the technique for aligning the extracted feature points to calculate the correspondence of the feature points, and a detailed explanation is omitted here.
[0074] In the case where there are n multiple multi-view images, the feature point extraction unit (410) in step S210 extracts each view image I in step S200. i The correspondence between each feature point can be calculated by matching the feature points of the remaining n-1 time frames with the feature points of the other n-1 time frames.
[0075] The camera geometric information estimation unit (420) estimates the camera geometric information of a plurality of multi-view images based on the correspondence of feature points calculated in step S220 (S220). In step S220), the camera geometric information can be estimated using one of the PnP technique, Bundle adjustment technique, Reduced Camera System technique, Conjugate Gradient technique, and SfM (Structure from Motion) technique. In addition, the technique for estimating the camera geometric information may utilize other techniques known in the art in addition to the aforementioned techniques, and a detailed description is omitted here.
[0076] In the case where there are n multiple multi-view images, in step S220, the camera geometric information estimation unit (420) is n multi-view images I i The camera geometric information can be estimated. That is, in step S220, the camera geometric information for each of the n multi-view images can be estimated. Here, the camera geometric information is multi-view image I i It may include the camera's location information at the time the image was taken.
[0077] The color value processing unit (430) calculates a tie point based on the camera geometric information estimated by the camera geometric information estimation unit (420), and stores the color value of a pixel corresponding to the tie point in the multi-view image used to calculate the tie point in association with the tie point (S230). The color value processing unit (430) can store the color value of the pixel corresponding to the tie point within the data structure of the corresponding tie point.
[0078] If there are n multiple multi-view images and m tie points are calculated in step S230, then among the m tie points in step S230, tie point t j When calculating each tie point t (where j represents a value between 1 and m), j The viewpoint image I used to calculate iand the video at that point in time I i The color value c of the pixel corresponding to ij tie point t j It can be stored inside a data structure. For example, t j Color value c stored inside the data structure ij It can be a color value of the RGB channel.
[0079] The channel conversion unit (320) converts the color value of the pixel stored in association with the tie point calculated in step S100 into a YCbCr channel (S110).
[0080] If there are n multiple multi-view images and m tie points are calculated in step S100, in step S110, the channel conversion unit (320) performs for each tie point t of the m tie points. j All color values c stored inside the data structure ij It can be converted to YCbCr channels.
[0081] The reference image selection unit (330) selects a reference image from among a plurality of multi-view images based on at least one of the luminance (Y) value among the color values converted by the channel conversion unit (320) and a preset rule (S120).
[0082] In step S120, the reference image selection unit (330) can classify the color values converted by the channel conversion unit (320) into each of a plurality of multi-view images to generate a set of color values for each multi-view image. Here, each set of color values may include the converted color values of the related multi-view image.
[0083] If there are n multiple multi-view images and m tie points are calculated in step S100, in step S120, the reference image selection unit (330) selects each tie point t for the m tie points. j All color values c stored inside the data structure ij is each multi-view image I i Classified as multi-view image I iThe set of color values C i It can generate.
[0084] In step S120, the reference image selection unit (330) can calculate the median of each color value in the generated set of color values.
[0085] When there are n multiple multi-view images and m tie points are calculated in step S100 and n sets of color values are calculated, the reference image selection unit (330) selects each set of color values C for the n sets of color values. i color value c in ij The median e i It can produce.
[0086] In step S120, the reference image selection unit (330) can select a reference image based on at least one of the calculated median value and a preset rule. The preset rule may be set by a user or selected by a creator.
[0087] In the case where there are n multiple multi-view images, the reference image selection unit (330) is reference image I t You can choose t (where t means a value between 1 and n).
[0088] In step S120, the reference image selection unit (330) can select a reference image from among a plurality of multi-view images based on the calculated median value. For n sets of color values, each set of color values C i color value c in ij The median e i In the case where this is calculated, the reference image selection unit (330) calculates the median value e i The viewpoint image that is the median of all multi-viewpoint images can be selected as the reference image.
[0089] The brightness correction graph calculation unit (340) calculates a brightness correction graph between a brightness reference image selected by the reference image selection unit (340) and another multi-view image based on the Y value (S130). In step S130, the brightness correction graph calculation unit (340) can calculate the brightness correction graph based on the correspondence relationship between the Y value of the brightness reference image and the Y value of another multi-view image based on each tie point. The Y value of the brightness reference image may be a Y value included in the set of color values of the reference image, and the Y value of the other view image may be a Y value included in the set of color values of the other view image.
[0090] Additionally, in step S130, the brightness correction graph calculation unit (340) may calculate the brightness correction graph using an error measurement method or a monotonically increasing third-order polynomial curve fitting technique. The error measurement method may include the Root Mean Square Error (RMSE) technique and other techniques known in the art, and a detailed description is omitted here.
[0091] There are n multiple multi-view images, m tie points are calculated in step S100, and brightness reference image I in step S120 t If selected, in step S130, the brightness correction graph calculation unit (340) calculates each tie point t for m tie points. j Brightness reference image I for t and target viewpoint image I i Brightness correction graph G using the correspondence relationship of Y values iIt can be calculated. At this time, the brightness correction graph calculation unit (340) can calculate the brightness correction graph by utilizing an error measurement method such as RMSE and existing techniques such as monotonically increasing third-order polynomial curve fitting. In step 130, the brightness correction graph calculation unit (340) calculates a brightness correction graph for each of the multi-view images, thereby producing n brightness correction graphs for n multi-view images.
[0092] The brightness correction unit (350) corrects the brightness of other multi-view images using the brightness correction graph calculated in step S130 (S140). In step S130, the brightness correction unit (350) can correct the brightness of the remaining multi-view images, excluding the reference image among the multiple multi-view images, based on the brightness correction graph calculated by the brightness correction graph calculation unit (340). Additionally, in step S140, the brightness correction unit (350) can convert the color values of the corrected multi-view images into RGB channels and store them.
[0093] In the case where n brightness correction graphs are calculated for n multi-view images in step S130, the brightness correction graph calculation unit (340) calculates each image I i Brightness correction graph G for i Using each image I i The brightness of the Y value can be corrected, and the corrected color value can be converted into RGB channels and saved.
[0094] FIG. 7 is a block diagram showing the configuration of a device for correcting the brightness of a multi-view image according to another embodiment of the present invention.
[0095] Referring to FIG. 7, a device (100) for correcting the brightness of a multi-view image may include a bus (101), a control unit (110), a storage unit (120), a user interface (UI) input device (150), a UI output device (160), and a communication unit (170). The storage unit (120) may include a memory (130) and a storage (140). The control unit (110), memory (130), storage (140), UI input device (150), UI output device (160), and communication unit (170) may communicate with each other through the bus (101). The device (100) for correcting the brightness of a multi-view image may be a computer system.
[0096] The control unit (110) may be a semiconductor device that executes processing instructions stored in memory (130) or storage (140). The control unit (110) may be at least one hardware processor. The control unit (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU), of a device (100) for correcting the brightness of a multi-view image. The control unit (110) may read a computer program loaded in memory (130) and perform data processing for learning a deep learning network according to an embodiment of the present invention.
[0097] The control unit (110) can execute commands and perform operations associated with the device (100) for correcting the brightness of a multi-view image. For example, using a command retrieved from the storage (140), the control unit (110) can control the input and output between components of the device (100) for correcting the brightness of a multi-view image, as well as the reception and processing of data.
[0098] The control unit (110) can perform the generation and processing of signals, data, or information that are input to the device (100) for correcting the brightness of a multi-view image, output from the device (100) for correcting the brightness of a multi-view image, or used internally within the device (100) for correcting the brightness of a multi-view image, and can perform inspection, comparison, and judgment related to the signals, data, or information. That is to say, in the embodiment, the generation and processing of data or information, and the inspection, comparison, and judgment related to the data or information can be performed by the control unit (110).
[0099] The control unit (110) can generate a bitstream. The control unit (110) can perform operations for generating a bitstream as described in the embodiments.
[0100] At least some of the parts, units, and modules described in the embodiments to be described below may be program modules and may communicate with an external device or system. The program modules may be included in the device (100) for correcting the brightness of a multi-view image in the form of an operating system, an application program module, and other program modules.
[0101] The program modules may be physically stored on various known memory devices. Additionally, at least some of these program modules may be stored in a remote memory device capable of communicating with a device (100) for correcting the brightness of a multi-view image.
[0102] Program modules may include, but are not limited to, routines, subroutines, programs, objects, program components, and data structures that perform functions or operations according to one embodiment or implement abstract data types according to one embodiment.
[0103] Program modules may consist of instructions or code executed by at least one processor of a device (100) for correcting the brightness of a multi-view image.
[0104] The control unit (110) can execute commands or codes of the units, units, and modules described in the embodiments.
[0105] The communication unit (170) can transmit and receive signals or data to and from the network (199) with a multi-view video providing server, other servers, and other terminals. The network (199) may be a private network or an internet network, and may include a wired network or a wireless network. Network (199) may refer to one or more parts of a network that may be an ad hoc network, intranet, extranet, Bluetooth, ZigBee, VPN (virtual private network), LAN (local area network), Wireless LAN (IEEE 802.11b, IEEE 802.11a, IEEE 802.11g, IEEE 802.11n), WIBro (Wireless Broadband), WAN (wide area network), WWAN (wireless WAN), MAN (metropolitan area network), Internet, part of the Internet, part of the PSTN (Public Switched Telephone Network), POTS (plain old telephone service) network, cellular telephone network, wireless network, Wi-Fi® network, other types of networks, or a combination of two or more of these networks, and may refer to one or more parts of a network to which other types of networks are connected. For example, a network or part of a network may include a wireless or cellular network, and a connection may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or another type of cellular or wireless connection.In this example, the connection may be implemented with any of various types of data transmission technologies, such as 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3GPP (third Generation Partnership Project) including 3G, 4G (fourth generation wireless) networks, UMTS (Universal Mobile Telecommunications System), HSPA (High Speed Packet Access), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution) standards, other things defined by various standards-setting organizations, other long-range protocols, or other data transmission technologies.
[0106] The communication unit (170) can transmit and receive signals or data with a server or other terminal that provides multi-view images using LAN, Wireless LAN (IEEE 802.11b, IEEE 802.11a, IEEE 802.11g, IEEE 802.11n), WIBro (Wireless Broadband), Bluetooth, and ZigBee.
[0107] Functions related to the communication of data or information of the device (100) for correcting the brightness of a multi-view image can be performed through the communication unit (170).
[0108] For example, the communication unit (170) can receive multiple multi-view images through the network (199). The communication unit (170) can receive multiple multi-view images from a camera, an electronic device, or a server.
[0109] The storage unit (120) may represent memory (130) and / or storage (140). Memory (130) and storage (140) may be various forms of volatile or non-volatile storage media. For example, memory (130) may include at least one of ROM (131) and RAM (132).
[0110] The storage unit (120) can store data or information used for the operation of the device (100) for correcting the brightness of a multi-view image. In an embodiment, the data or information of the device (100) for correcting the brightness of a multi-view image can be stored in the storage unit.
[0111] The recording medium may store at least one module required for the device (100) for correcting the brightness of a multi-view image to operate. The storage (140) or ROM (131) may store at least one module. The at least one module stored in the recording medium may be configured to be accessed from the recording medium by the control unit (110), loaded into RAM (132), and executed.
[0112] The storage unit (140) can store multiple multi-view images. Here, the multiple multi-view images stored in the storage unit (140) may be images input from at least one camera, images input by a user, and images received by the communication unit (170). Additionally, the multiple multi-view images may be images captured in the manner described in FIG. 1 or FIG. 2.
[0113] The UI input device (150) may be configured to receive input data and transmit it to the control unit (110). The UI input device (150) may receive multiple multi-view images from a user or a camera.
[0114] The UI output device (160) can be configured to output the processing result of the control unit (110).
[0115] The control unit (110) can implement the methods described based on the aforementioned FIGS. 5 and FIGS. 6. Additionally, the control unit (110) may be configured to perform the functions of a tie point calculation unit (310), a channel conversion unit (320), a reference image selection unit (330), a brightness correction graph calculation unit (340), and a brightness correction unit (350). That is, the control unit (110) can load a plurality of multi-view images into RAM (110) and perform the functions of a tie point calculation unit (310), a channel conversion unit (320), a reference image selection unit (330), a brightness correction graph calculation unit (340), and a brightness correction unit (350) for the loaded plurality of multi-view images. Here, the plurality of multi-view images may include a plurality of multi-view images stored in a storage unit (140), a plurality of multi-view images received by a communication unit (130), or a plurality of multi-view images received by a UI input device (150).
[0116] In the embodiments described above, when applying a specific treatment to a specific target, a specific condition may be required, and if it is described that the specific treatment is performed under a specific decision, and if it is described that whether the specific condition is satisfied is determined based on a specific coding parameter, or that a specific decision is made based on a specific coding parameter, the specific coding parameter may be interpreted as being replaceable with another coding parameter. That is to say, the coding parameter affecting the specific condition or the specific decision may be considered merely exemplary, and it may be understood that a combination of one or more other coding parameters in addition to the specified coding parameter performs the role of the specified coding parameter.
[0117] In the embodiments described above, methods are described based on flowcharts as a series of steps or units; however, the present invention is not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps as described above. Furthermore, those skilled in the art will understand that the steps shown in the flowcharts are not exclusive, that other steps may be included, or that one or more steps of the flowcharts may be omitted without affecting the scope of the present invention.
[0118] The embodiments described above include examples of various aspects. While not all possible combinations for representing various aspects can be described, those skilled in the art will recognize that combinations other than those explicitly described are possible. Accordingly, the present invention shall be deemed to include all other substitutions, modifications, and changes falling within the scope of the following claims.
[0119] The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software.
[0120] A computer-readable recording medium may include information used in embodiments according to the present invention. For example, a computer-readable recording medium may include a bitstream, and the bitstream may include information described in embodiments according to the present invention.
[0121] The bitstream may include computer-executable code and / or programs. The computer-executable code and / or programs may include information described in the embodiments and syntax elements described in the embodiments. That is to say, the information and syntax elements described in the actual examples may be considered as computer-executable code within the bitstream and may be considered as at least part of the computer-executable code and / or programs represented by the bitstream.
[0122] Computer-readable recording media may include non-transitory computer-readable media.
[0123] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The above hardware devices may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.
[0124] Although the present invention has been described above with specific details such as specific components, limited embodiments, and drawings, this is provided only to aid in a more comprehensive understanding of the invention, and the invention is not limited to the above embodiments, and a person skilled in the art to which the invention belongs can make various modifications and variations from this description.
[0125] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the claims set forth below, as well as the claims described below, shall be considered to fall within the scope of the concept of the present invention.
Claims
Claim 1 A method for correcting the brightness of a multi-view image, comprising: a step of calculating a tie point in a plurality of multi-view images and storing the color value of a pixel corresponding to the tie point in the multi-view image used for calculating the tie point in association with the tie point; a step of converting the stored color value of the pixel into a YCbCr channel; a step of selecting a reference image among the plurality of multi-view images based on at least one of a luminance (Y) value among the converted color values and a pre-set rule; a step of calculating a brightness correction graph between the selected reference image and another multi-view image based on the Y value; and a step of correcting the brightness of the other multi-view image using the calculated brightness correction graph. Claim 2 A method for correcting the brightness of a multi-view image according to claim 1, wherein the step of storing in association with the tie point comprises: a step of extracting feature points from the plurality of multi-view images and aligning the extracted feature points to calculate the correspondence relationship of the feature points; a step of estimating camera geometric information of the plurality of multi-view images based on the calculated correspondence relationship; and a step of calculating the tie point based on the estimated camera geometric information and storing the color value of a pixel corresponding to the tie point in the multi-view image used when calculating the tie point in association with the tie point. Claim 3 In claim 1, the color value of the pixel corresponding to the tie point is a method for correcting the brightness of a multi-view image stored within the data structure of the tie point. Claim 4 A method for correcting the brightness of a multi-view image according to claim 1, wherein the step of selecting the reference image comprises: classifying the converted color values into each of the plurality of multi-view images to generate a set of color values for each multi-view image; calculating the median of each color value in the generated set of color values; and selecting the reference image based on the calculated median. Claim 5 In claim 1, the above-mentioned pre-set rule is a method for correcting the brightness of a multi-view image set by a user. Claim 6 A method for correcting the brightness of a multi-view image, wherein the step of calculating the brightness correction graph in claim 1 comprises the step of calculating the brightness correction graph based on the correspondence relationship between the Y value of the reference image and the Y value of the other multi-view image. Claim 7 A method for correcting the brightness of a multi-view image, wherein the step of calculating the brightness correction graph comprises the step of calculating the brightness correction graph using an error measurement method or a monotonically increasing third-order polynomial curve fitting technique. Claim 8 A method for correcting the brightness of a multi-view image according to claim 1, wherein the step of correcting the brightness of the multi-view image comprises the step of converting the color value of the corrected multi-view image into RGB channels and storing it. Claim 9 In claim 2, the above feature points are a method for correcting the brightness of a multi-view image extracted from each multi-view image using a Scale Invariant Feature Transform (SIFT) technique. Claim 10 In claim 2, the camera geometric information is a method for correcting the brightness of a viewpoint image estimated using one of the PnP technique, Bundle adjustment technique, Reduced Camera System technique, Conjugate Gradient technique, and SfM (Structure from Motion) technique. Claim 11 An apparatus for correcting the brightness of a multi-view image, comprising: a storage unit for storing a plurality of multi-view images; and a control unit for calculating a tie point in the plurality of multi-view images, associating the color value of a pixel corresponding to the tie point in the multi-view image used when calculating the tie point with the tie point and storing it in the storage unit, converting the stored color value of the pixel into a YCbCr channel, selecting a reference image among the plurality of multi-view images based on at least one of a luminance (Y) value among the converted color values and a preset rule, calculating a brightness correction graph between the selected reference image and another multi-view image using the Y value, and correcting the brightness of the other multi-view image using the calculated brightness correction graph. Claim 12 A device for correcting the brightness of a multi-view image, wherein the control unit extracts feature points from the plurality of multi-view images, aligns the extracted feature points to calculate the correspondence relationship of the feature points, estimates camera geometric information of the plurality of multi-view images based on the calculated correspondence relationship, calculates the tie point based on the estimated camera geometric information, and stores the color value of a pixel corresponding to the tie point in the multi-view image used when calculating the tie point in association with the tie point in the storage unit. Claim 13 In claim 11, the color value of the pixel corresponding to the tie point is a device for correcting the brightness of a multi-view image stored within the data structure of the tie point. Claim 14 In claim 11, the control unit classifies the converted color values into each of the plurality of multi-view images to generate a set of color values for each multi-view image, calculates a median value for each color value in the generated set of color values, and selects a reference image based on the calculated median value, for a device for correcting the brightness of a multi-view image. Claim 15 In Clause 11, the above-mentioned pre-set rule is a device for correcting the brightness of a multi-view image set by a user. Claim 16 In claim 11, the control unit is a device for correcting the brightness of a multi-view image, which calculates the brightness correction graph based on the correspondence relationship between the Y value of the reference image and the Y value of the other multi-view image. Claim 17 In claim 16, the control unit is a device for correcting the brightness of a multi-view image, which calculates the brightness correction graph using an error measurement method or a monotonically increasing third-order polynomial curve fitting technique. Claim 18 In claim 11, the control unit is a device for correcting the brightness of a multi-view image, which converts the color value of the corrected multi-view image into RGB channels and stores it in the storage unit. Claim 19 In claim 12, the above feature points are a method for correcting the brightness of a multi-view image extracted from each multi-view image using a Scale Invariant Feature Transform (SIFT) technique. Claim 20 A computer program stored on a computing device-readable medium to execute the method described in any one of claims 1 through 10.