RGB-IR sensor calibration method and device, storage medium and electronic device

Abstract

The embodiment of the invention provides an RGB-IR sensor calibration method and device, a storage medium and an electronic device, and the method comprises the steps: obtaining a first calibration matrix containing IR signal residues and a second calibration matrix which filters IR signal residues according to a filtered IR signal of a to-be-calibrated scene and a to-be-calibrated signal containing the IR signal; and obtaining an optimized calibration matrix according to the first calibration matrix and the second calibration matrix so as to calibrate the RGB-IR sensor. Through the method and the device, the problems of poor calibration accuracy and stability and low scene color rendition correctness of the IR filtering filter in related technologies are solved, and the effects of improving the calibration accuracy and stability of the IR filtering filter and improving the scene color rendition correctness are achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a RGB-IR sensor calibration method and device, a storage medium, and an electronic device. Background Art

[0002] RGB-IR sensor is a sensor that integrates red, green, blue and infrared (IR) sensing functions. Traditional RGB-based sensors can only capture the color information of objects. RGB-IR sensors can obtain the distance information of objects by effectively utilizing IR signals while measuring the color of objects. In addition, in low-light scenes, the sensitivity of capturing objects through IR spectrum is higher than that of RGB spectrum, which can effectively improve the recognition of objects in low light. Based on the above advantages, RGB-IR sensors are widely used in smart homes, smart security, autonomous driving and other fields.

[0003] The RGB-IR sensor receives IR and RGB signals at the same time, which causes crosstalk between the color signal and the infrared signal. How to effectively filter out the IR signal in the RGB signal greatly affects the accuracy of the scene color reproduction. The industry usually uses a calibration method to obtain the IR filter, that is, by shooting a standard scene with an IR signal and completely filtering out the IR signal, the desired IR filter is obtained through a digital processing algorithm. To obtain the calibration signal for filtering out the IR signal, it is necessary to use an infrared cutoff (IR-CUT) filter to ensure that the IR signal is completely filtered out. However, in practice, due to cost and process limitations, the IR-CUT stopband cutoff characteristics are insufficient, resulting in residual IR signals being mixed in the RGB signal, thereby reducing the calibration accuracy and stability of the IR filter, and ultimately affecting the correctness of the scene color reproduction. Summary of the invention

[0004] The embodiment of the present invention provides a RGB-IR sensor calibration method and device to at least solve the problems of poor calibration accuracy and stability of IR rejection filters and low correctness of scene color reproduction in the related art.

[0005] According to one embodiment of the present invention, a RGB-IR sensor calibration method is provided, comprising: obtaining a first calibration matrix containing IR signal residues and a second calibration matrix filtering out the IR signal residues based on filtered infrared IR signals of a scene to be calibrated and a signal to be calibrated containing IR signals; obtaining an optimized calibration matrix based on the first calibration matrix and the second calibration matrix to calibrate the RGB-IR sensor.

[0006] According to another embodiment of the present invention, a RGB-IR sensor calibration device is provided, including: a calibration acquisition module, used to acquire a first calibration matrix containing IR signal residues and a second calibration matrix filtering out the IR signal residues based on the filtered infrared IR signal of the scene to be calibrated and the signal to be calibrated containing the IR signal; a calibration optimization module, used to acquire an optimized calibration matrix based on the first calibration matrix and the second calibration matrix to calibrate the RGB-IR sensor.

[0007] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0008] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0009] The present invention provides a calibration method for an RGB-IR sensor, wherein a first calibration matrix containing residual IR signals and a second calibration matrix containing residual IR signals are obtained according to the filtered infrared IR signals of the scene to be calibrated and the signal to be calibrated containing IR signals; an optimized calibration matrix is ​​obtained according to the first calibration matrix and the second calibration matrix to calibrate the RGB-IR sensor. The method solves the problems of poor calibration accuracy and stability of the IR filter and low accuracy of scene color restoration in the related art, and achieves the effect of improving the calibration accuracy and stability of the IR filter and improving the accuracy of scene color restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a hardware structure block diagram of a computer terminal of an RGB-IR sensor calibration method according to an embodiment of the present invention;

[0011] Figure 2 is a flow chart of a RGB-IR sensor calibration method according to an embodiment of the present invention;

[0012] Figure 3 is a flow chart of a RGB-IR sensor calibration method according to an embodiment of the present invention;

[0013] Figure 4 is a structural block diagram of an RGB-IR sensor calibration device according to an embodiment of the present invention;

[0014] Figure 5 is a structural block diagram of an RGB-IR sensor calibration device according to an embodiment of the present invention;

[0015] Figure 6 is a schematic diagram of the RGB-IR format according to a scenario embodiment of the present invention;

[0016] Figure 7 is a flow chart of a RGB-IR sensor calibration method according to a scenario embodiment of the present invention. DETAILED DESCRIPTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a computer terminal of an RGB-IR sensor calibration method according to an embodiment of the present invention. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0020] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the RGB-IR sensor calibration method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0021] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0022] In this embodiment, a RGB-IR sensor calibration method running on the above-mentioned computer terminal is provided. Figure 2 is a flow chart of a RGB-IR sensor calibration method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0023] Step S202, according to the filtered infrared IR signal of the scene to be calibrated and the signal to be calibrated containing the IR signal, a first calibration matrix containing the IR signal residue and a second calibration matrix containing the filtered IR signal residue are obtained.

[0024] In the embodiment of the present invention, the IR-filtered signal is a signal to be calibrated that does not contain an IR signal. The signal to be calibrated that contains an IR signal is a signal to be calibrated that has not had IR filtered out, and may also be referred to as an original signal. The IR-filtered signal may be obtained by performing infrared filtering on the signal to be calibrated that contains an IR signal.

[0025] In an exemplary embodiment, a first calibration matrix containing IR signal residues is obtained based on the filtered infrared IR signal of the scene to be calibrated and the signal to be calibrated containing the IR signal, including: according to the signal to be calibrated and the filtered IR signal, a least squares method or a convex optimization method is used to obtain the first calibration matrix containing IR signal residues.

[0026] In an exemplary embodiment, based on the filtered infrared IR signal of the scene to be calibrated and the signal to be calibrated containing the IR signal, a second calibration matrix with the IR signal filtered out is obtained, including: obtaining a white block signal of the scene to be calibrated; and based on the white block signal, using a least squares method or a convex optimization method to obtain a second calibration matrix.

[0027] In actual implementation, the method of obtaining the first calibration matrix and the second calibration matrix is ​​not limited to the least square method or the convex optimization method, and the present invention will not give examples one by one.

[0028] In an exemplary embodiment, the method for obtaining the expected white balance coefficient of the white block signal includes at least one of the following: estimating the expected white balance coefficient based on the white balance coefficient reference value of the G channel; obtaining the expected white balance coefficient using a white balance coefficient calibration method; obtaining the standard G channel brightness value through a color card to obtain the expected white balance coefficient.

[0029] Step S204: acquiring an optimized calibration matrix according to the first calibration matrix and the second calibration matrix to calibrate the RGB-IR sensor.

[0030] In an exemplary embodiment, obtaining an optimized calibration matrix according to the first calibration matrix and the second calibration matrix includes: performing a simultaneous operation on the first calibration matrix and the second calibration matrix to obtain the optimized calibration matrix.

[0031] In an exemplary embodiment, before step 202, the method further includes: acquiring the filtered IR signal and the signal to be calibrated of the scene to be calibrated; and preprocessing the filtered IR signal and the signal to be calibrated to obtain the filtered IR signal and the signal to be calibrated after preprocessing.

[0032] In actual implementation, the preprocessing step can be omitted.

[0033] In an exemplary embodiment, obtaining the filtered IR signal and the signal to be calibrated of the scene to be calibrated includes: according to the scene to be calibrated, obtaining the signal to be calibrated through an RGB-IR sensor, and obtaining the filtered IR signal through an infrared cutoff IR-cut.

[0034] In an exemplary embodiment, before obtaining a first calibration matrix including IR signal residues and a second calibration matrix after filtering out IR signal residues, the method further comprises: calculating multi-channel means of the signal to be calibrated and the IR signal after filtering out for each of the plurality of blocks.

[0035] In actual implementation, the step of obtaining the multi-channel mean can be omitted.

[0036] Figure 3 is a flow chart of a RGB-IR sensor calibration method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0037] Step S302, obtaining the filtered IR signal and the signal to be calibrated of the scene to be calibrated;

[0038] Step S304, preprocessing the filtered IR signal and the signal to be calibrated to obtain the filtered IR signal and the signal to be calibrated after preprocessing;

[0039] Step S306, for multiple blocks, respectively calculating the multi-channel mean of the signal to be calibrated and the filtered IR signal;

[0040] Step S308, obtaining a first calibration matrix including the IR signal residue and a second calibration matrix after filtering out the IR signal residue according to the IR signal filtered out and the signal to be calibrated;

[0041] Step S310: Obtain an optimized calibration matrix according to the first calibration matrix and the second calibration matrix to calibrate the RGB-IR sensor.

[0042] In actual implementation, in step S308, the first calibration matrix and the second calibration matrix may be obtained based on the multi-channel mean of the signal to be calibrated and the filtered IR signal; or the first calibration matrix and the second calibration matrix may be obtained directly based on the filtered IR signal and the signal to be calibrated.

[0043] Through the above steps, a first calibration matrix containing IR signal residues and a second calibration matrix containing IR signal residues are obtained according to the filtered infrared IR signal of the scene to be calibrated and the signal to be calibrated containing IR signal; an optimized calibration matrix is ​​obtained according to the first calibration matrix and the second calibration matrix to calibrate the RGB-IR sensor. Through the present invention, the problems of poor calibration accuracy and stability of IR filter and low scene color restoration accuracy in the related art are solved, and the effect of improving the calibration accuracy and stability of IR filter and improving the scene color restoration accuracy is achieved.

[0044] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0045] In this embodiment, a RGB-IR sensor calibration device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0046] Figure 4 is a structural block diagram of an RGB-IR sensor calibration device according to an embodiment of the present invention. Figure 4 As shown, the calibration device 40 includes: a calibration acquisition module 410, which is used to obtain a first calibration matrix containing IR signal residues and a second calibration matrix containing IR signal residues after filtering out the IR signal of the scene to be calibrated and the signal to be calibrated containing the IR signal; a calibration optimization module 420, which is used to obtain an optimized calibration matrix according to the first calibration matrix and the second calibration matrix to calibrate the RGB-IR sensor.

[0047] In an exemplary embodiment, the calibration acquisition module 410 uses the least square method or convex optimization method to obtain a first calibration matrix containing the IR signal residue according to the signal to be calibrated and the filtered IR signal. In an exemplary embodiment, the calibration acquisition module 410 obtains the white block signal of the scene to be calibrated; according to the white block signal, uses the least square method or convex optimization method to obtain a second calibration matrix.

[0048] In an exemplary embodiment, the calibration optimization module 420 performs a joint operation on the first calibration matrix and the second calibration matrix to obtain an optimized calibration matrix.

[0049] In an exemplary embodiment, the calibration device also includes: a signal acquisition module, used to acquire the filtered IR signal and the signal to be calibrated of the scene to be calibrated; a signal processing module, used to preprocess the filtered IR signal and the signal to be calibrated to obtain the filtered IR signal and the signal to be calibrated after preprocessing.

[0050] In an exemplary embodiment, the signal acquisition module acquires the signal to be calibrated through the RGB-IR sensor according to the scene to be calibrated, and acquires the filtered IR signal through the infrared cutoff IR-cut.

[0051] In an exemplary embodiment, the calibration device further includes: a mean value acquisition module, which is used to calculate the multi-channel mean values ​​of the signal to be calibrated and the filtered IR signal for each of the multiple blocks.

[0052] Figure 5 is a structural block diagram of an RGB-IR sensor calibration device according to an embodiment of the present invention. According to the above embodiment, Figure 5 As shown, the calibration device 50 includes: a signal acquisition module 510, a signal processing module 520, a mean acquisition module 530, a calibration acquisition module 540, and a calibration optimization module 550.

[0053] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0054] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0055] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0056] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0057] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0058] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0059] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0060] In order to enable those skilled in the art to better understand the technical solution of the present invention, it is described below in conjunction with specific scenario embodiments.

[0061] Scenario Example 1

[0062] The scenario embodiment of the present invention aims at the calibration problem of the IR filter during the use of the RGB-IR sensor, and proposes a calibration method for the RGB-IR sensor based on the IR filter optimized by the IR residue. The calibration process is decomposed to calibrate the filter that can characterize the IR residue and the filter that filters out the IR residue, and finally the optimized IR filter is obtained by combining the method, which can effectively improve the accuracy of the IR filter calibration and improve the accuracy of the scene color restoration. Among them, the above-mentioned IR filter can be understood as a filter or calibration signal or calibration matrix for IR filtering. The filter that characterizes the IR residue can be understood as a residual IR signal after the original signal to be calibrated is infrared filtered, wherein the signal that can characterize the IR residue is an IR residue filter, which can also be called an IR residue signal or an IR residue calibration signal. The filter that filters out the IR residue is further filtered for the IR residue, and the filter that filters out the IR residue can also be called a filter IR residue calibration signal or a filter IR residue calibration matrix. The optimized IR filter is the optimized filter finally obtained, which can also be called an optimized calibration filter or an optimized calibration matrix.

[0063] Figure 6 is a schematic diagram of the RGB-IR format according to a scenario embodiment of the present invention, such as Figure 6 As shown, RGB-IR sensors are generally divided into two formats: RGB-IR 2x2 and 4x4. RGB-IR sensors can simultaneously obtain the color information of an object (contained in the RGB signal) and the IR signal reflected by the object through a spatial downsampling method. The technical solution proposed in the embodiment of the present invention can also be applied to other RGB-IR format variants, such as different arrangements and combinations of RGB colors and IR channels in spatial positions.

[0064] Figure 7 is a flow chart of a RGB-IR sensor calibration method according to a scenario embodiment of the present invention. Figure 7 As shown, the following steps are included:

[0065] Step S701, for the same calibration scene, respectively using and not using infrared cutoff IR-cut, obtaining RGB-IR receiving signals, and performing pixel level interpolation.

[0066] For a set of calibration scenes, the IR signal and the calibration images of the filtered IR signal can be obtained by directly using the RGB-IR sensor and adding an infrared cutoff IR-cut inside or outside the module (i.e., the signal to be calibrated and the filtered infrared IR signal in the above embodiment).

[0067] First, define a block consisting of adjacent 4x4 pixels, and the received signal is:

[0068]

[0069] The subscripts correspond to Figure 6 The spatial position of the pixel in the RGB-IR 4x4. For any pixel, its spatial position index is defined as [i, j], where [i, j] represents the subscript. The pixel value P of the four channels [R, G, B, IR] at this pixel position can be obtained by interpolation method. i,j ,

[0070] P i,j =[R i,j ,G i,j ,B i,j ,IR i,j ] T

[0071] Among them, the bold P represents a 4x1 dimension vector, [.] T The optional interpolation methods include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation and the like, and the embodiment of the present invention does not limit which interpolation method is used.

[0072] Get IR signal P i,j and filter out IR signal Ps i,j for:

[0073] P i,j =[R i,j ,G i,j ,B i,j ,IR i,j ] T

[0074] Ps i,j =[Rs i,j ,Gs i,j ,Bs i,j ,IRs i,j ] T

[0075] Step S702: IR signal P i,j and filter out IR signal Ps i,j Perform preprocessing.

[0076] The signal P can be processed by preprocessing method i,j and Ps i,j , and get Q i,j and Qs i,j ,

[0077]

[0078]

[0079] Among them, [gain r ,gain g ,gain b ,gain ir ] indicates the adjustment gain of the corresponding channel, [offset r ,offset g ,offset b ,offset ir ] indicates adjusting the bias. In actual implementation, preprocessing may not be performed. If no preprocessing method is required, the gain may be set to 1 and the bias may be set to 0.

[0080] When Qs i,j When the IR residue is negligible, the desired IR removal filter is defined as a 3x4 matrix W 3×4 ,have

[0081]

[0082] in, Indicates Qs i,j Remove the IR dimension.

[0083] Step S703, calculating the multi-pixel channel mean at the block level.

[0084] For the stability of the calibration scheme, multiple blocks are usually used to perform the above W 3×4 Calibration, define block index as m, m = 0, 1, 2, ..., M, there are,

[0085]

[0086] in, represents the mean value of the R channel in the block m area, that is The same applies to other cases.

[0087] In actual implementation, the above process of obtaining the average value can be omitted.

[0088] In the actual implementation process, W is obtained by the least squares method (LS) or convex optimization method. 3×4 , for example using the LS method,

[0089]

[0090] Where pinv(.) represents the matrix pseudo-inverse.

[0091] Through the above steps, in Qs i,j When the IR residue in W is small, the correct IR filter matrix can usually be obtained. Due to process and cost limitations, IR-cut usually does not have a perfect stopband cutoff characteristic. That is, even if IR-cut is used, IR residue usually still exists in the RGB calibration image that is expected to not contain IR, which leads to the IR filter matrix W 3×4 The calibration is inaccurate. For such a problem, the embodiment of the present invention solves it through the following steps.

[0092] Step S704, calculate the calibration matrix Ws representing the IR residual 4×4 .

[0093] W 4×4 The IR filter matrix representing the IR residual effect, i.e., the first calibration matrix in the above embodiment,

[0094]

[0095] Among them, the fourth column of the matrix on the left side of the equal sign can represent the IR residual impact.

[0096] Ws can be calculated by LS or convex optimization method 4×4 , for example using the LS method:

[0097]

[0098] In the embodiment of the present invention, Ws 4×4 The calculation method is specifically restricted.

[0099] Step S705: Calculate the calibration matrix Ws that can filter out the IR residue 3×4 .

[0100] The white block prior information can be obtained by using a standard calibration color card during the calibration process, determining the position of the white block in the calibration image, and obtaining the white block [R, G, B] signal in the calibration image. The white block index is n = 0, 1, ..., N.

[0101]

[0102] Where Ws 3×4 The calibration matrix for filtering out the IR residual effect is the second calibration matrix in the above embodiment. The left side of the equation represents the desired white block signal, and the matrix on the right side represents the actually obtained white block signal. Each row of the equation can use the pixel mean or pixel level value of the same channel of the white block.

[0103] For the above desired white block signal,

[0104]

[0105] where [gr,gg,gb] T Represents the inverse of the desired white balance coefficient.

[0106] Combining the above equations, we can get:

[0107]

[0108] Among them, the expected white balance coefficients [gr, gg, gb] can be expressed using the estimated white balance coefficients:

[0109]

[0110]

[0111] gg=1

[0112] Among them, mean(.) means to find the mean of the vector, and n represents the element index of the vector for which the mean is to be found.

[0113] Or, obtain it through other white balance coefficient calibration methods. G channel reference value The following estimates can be used,

[0114]

[0115] Or, get the standard G channel brightness value through the color card.

[0116] For the convenience of formula description, the matrix is ​​rewritten in this scenario embodiment to simplify the matrix:

[0117]

[0118]

[0119] Ws can be obtained based on LS or convex optimization theory 3×4 , for example, for the LS method,

[0120] W 3×4 =L·pinv(R)

[0121] When the number of white blocks in the calibration color card is insufficient, that is, N is small, Ws can be calculated more accurately by using the convex optimization method. 3×4 , to alleviate the matrix rank deficiency problem, a convex optimization objective function can be expressed as follows:

[0122]

[0123]

[0124] where ||.|| 2 represents the matrix 2 norm, w represents the variable matrix, argmin() represents the minimum value, [c0,c1] represents the configurable coefficient, and loss represents the loss function. Ws can be obtained by an optional convex optimization method. 3×4 , such as Newton iteration method, gradient descent method, etc. The present invention does not limit the convex optimization method used, or the convex optimization objective function setting, and the number of hyperparameters.

[0125] Note that the above loss function can be extended to other forms. The present invention does not limit the specific loss function design method. An example of an extension method is as follows:

[0126]

[0127] Step S706: Ws 3×4 and Ws 4×4 Combine to obtain the optimized calibration matrix W opt .

[0128] W opt =Ws 3×4 ·Ws 4×4 ,

[0129] Among them, W opt Ws represents the desired IR filter matrix, that is, the optimized calibration matrix in the above embodiment. 3×4 represents the IR residual filter matrix, Ws 4×4 represents the calibration matrix that can characterize the IR residual.

[0130] In summary, the present invention aims at the problem of inaccurate calibration of the IR rejection filter caused by IR residue during the use of the RGB-IR sensor, and proposes a high-precision IR rejection filter calibration method. RGB-IR sensors have the characteristics of being able to simultaneously present color and object depth, as well as high recognition under extremely low illumination, and are widely used in smart homes, autonomous driving, and smart solutions. The present invention effectively solves the problem of inaccurate calibration of the IR rejection filter during the use of RGB-IR sensors, and can be used accordingly in the above-mentioned fields. The present invention proposes to calibrate the calibration matrix Ws that can characterize the IR residue in two steps. 4×4 , and the calibration matrix Ws that can filter out the IR residue 3×4 , and then combine to obtain the calibration matrix W based on IR residual optimization opt The method can effectively improve the calibration accuracy of the IR filter. The present invention provides a calibration matrix Ws for filtering out IR residues. 3×4The specific method of effectively utilizing the prior information of white blocks and combining it with the convex optimization theory can effectively solve the problem of insufficient stability caused by the lack of matrix rank during the calibration process. The method proposed in the present invention can effectively solve the problem of inaccurate calibration of IR filter due to IR-cut process limitations, and can effectively improve the accuracy of scene color restoration.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A RGB-IR sensor calibration method, characterized in that: include: According to the filtered infrared IR signal of the scene to be calibrated and the signal to be calibrated containing the IR signal, a first calibration matrix containing the IR signal residue and a second calibration matrix in which the IR signal residue is filtered out are obtained; An optimized calibration matrix is ​​obtained according to the first calibration matrix and the second calibration matrix to calibrate the RGB-IR sensor.

2. The method according to claim 1, characterized in that The step of obtaining a first calibration matrix including IR signal residues according to the filtered infrared IR signal of the scene to be calibrated and the signal to be calibrated including the IR signal comprises: According to the signal to be calibrated and the filtered IR signal, a least square method or a convex optimization method is used to obtain the first calibration matrix containing the IR signal residue.

3. The method according to claim 1, characterized in that The step of obtaining a second calibration matrix after filtering out the residual infrared IR signal of the scene to be calibrated and the signal to be calibrated containing the IR signal comprises: Acquire a white block signal of the scene to be calibrated; According to the white block signal, the second calibration matrix is ​​obtained by adopting a least square method or a convex optimization method.

4. The method according to claim 3, characterized in that in, The method of obtaining the expected white balance coefficient of the white block signal includes at least one of the following: Estimating and acquiring the desired white balance coefficient according to the white balance coefficient reference value of the G channel; Acquire the desired white balance coefficient using a white balance coefficient calibration method; The standard G channel brightness value is obtained through the color card to obtain the desired white balance coefficient.

5. The method according to claim 1, characterized in that: Acquiring an optimized calibration matrix according to the first calibration matrix and the second calibration matrix includes: A joint operation is performed on the first calibration matrix and the second calibration matrix to obtain the optimized calibration matrix.

6. The method according to claim 1, characterized in that Before acquiring the first calibration matrix including the IR signal residue and the second calibration matrix filtering out the IR signal residue, the method further includes: Acquire the filtered IR signal and the signal to be calibrated of the scene to be calibrated; The filtered IR signal and the signal to be calibrated are preprocessed to obtain the filtered IR signal and the signal to be calibrated after the preprocessing.

7. The method according to claim 6, characterized in that Acquiring the filtered IR signal and the signal to be calibrated of the scene to be calibrated, comprising: According to the scene to be calibrated, the signal to be calibrated is acquired through the RGB-IR sensor, and the filtered IR signal is acquired through infrared cutoff IR-cut.

8. The method according to claim 1, characterized in that: Before acquiring the first calibration matrix including the IR signal residue and the second calibration matrix filtering out the IR signal residue, the method further includes: For a plurality of blocks, multi-channel averages of the signal to be calibrated and the filtered IR signal are calculated respectively.

9. A RGB-IR sensor calibration device, characterized in that: include: A calibration acquisition module, used for acquiring a first calibration matrix containing IR signal residues and a second calibration matrix filtering out the IR signal residues according to the filtered infrared IR signal of the scene to be calibrated and the signal to be calibrated containing the IR signal; The calibration optimization module is used to obtain an optimized calibration matrix according to the first calibration matrix and the second calibration matrix to calibrate the RGB-IR sensor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the method described in any one of claims 1 to 8 when executed by a processor.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method described in any one of claims 1 to 8 is implemented.

Images