Method and device for correcting defective pixel of image

[0077] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0078] The embodiments of the present invention provide a method and device for correcting image dead pixels, which are used to solve the problem of low accuracy of bad pixel correction in the prior art. Among them, the method and the device are based on the same inventive concept. Since the principles of the method and the device to solve the problem are similar, the implementation of the device and the method can be referred to each other, and the repetition will not be repeated.

[0079] In order to make the embodiments of the present invention easier to understand, the following first describes some descriptions involved in the embodiments of the present invention, which should not be regarded as limiting the scope of protection required by the present invention.

[0080] Bayer image: The image sensor is usually covered with a filter called "color filter array (English: color filter array, monitoring and function: CFA)", which is generally a Bayer filter. The image collected by the Bayer filter is the Bayer image. Each pixel in the Bayer image has only one color information, and the pixels of the same color are distributed in 1 row or 1 column, such as figure 1 Shown.

[0081] Same color pixels: the pixels in the Bayer image that have the same color as the pixels to be detected. figure 1 In the Bayer image shown, R33 is a pixel to be detected as an example, and pixels such as R11, R13, R15, ... are pixels of the same color.

[0082] Heterochromatic pixels: the pixels in the Bayer image with different colors from the pixels to be detected, to figure 1 The R33 in the Bayer image shown is the pixel to be detected as an example, and the pixels such as B00, B44, G12, G25, ... are heterochromatic pixels.

[0083] Edge pixels: Refers to the pixels on both sides of a certain pixel in the image to be detected whose gray values ​​are significantly different, then the pixel is an edge pixel.

[0084] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0085] See figure 2 As shown, it is a flowchart of a method for correcting image dead pixels provided by an embodiment of the present invention. The method can be applied to electronic devices, including but not limited to: image sensors, image processors, image acquisition devices, etc. Etc., the method may specifically include the following:

[0086] S201: For the image to be detected, the electronic device establishes an N*N pixel matrix with the f-th pixel in the image to be detected as a central pixel point.

[0087] The f is a positive integer not greater than the total number of pixels included in the image to be detected; the N is an odd number greater than 1. The fth pixel is the pixel to be detected.

[0088] The image to be detected may be a Bayer image. To figure 1 In the Bayer image shown, R33 is the central pixel as an example. When N is 7, the 7*7 pixel matrix is ​​as image 3 Shown.

[0089] Optionally, before establishing an N*N pixel matrix with the f-th pixel as the central pixel, the electronic device performs mirror-flanging processing on the upper boundary, lower boundary, left boundary, and right boundary of the image to be detected, respectively.

[0090] S202: The electronic device converts the color of each pixel of the inner circle and the outer circle into the color of the central pixel, and determines the converted pixel value of each pixel.

[0091] The inner circle is a pixel point adjacent to the central pixel point in the N*N pixel matrix; the outer circle is a line spaced from the central pixel point in the N*N pixel matrix and/ Or b pixels in a row; the a and b are integers greater than one.

[0092] To image 3 The 7*7 pixel matrix shown as an example, when a is 8, b is 16, the distribution diagram of the inner circle and the outer circle is as follows Figure 4 Shown.

[0093] S203: The electronic device determines that the central pixel is a dead pixel based on the converted pixel value of each pixel, and corrects the central pixel.

[0094] In the embodiment of the present invention, for the image to be detected, the electronic device establishes an N*N pixel matrix with the f-th pixel in the image to be detected as the central pixel; the electronic device divides each pixel of the inner circle and the outer circle The color of the dot is converted into the color of the central pixel, and the converted pixel value of each pixel is respectively determined; the electronic device determines the central pixel based on the converted pixel value of each pixel The dot is a bad dot, and the center pixel dot is corrected. Compared with the method of detecting and correcting the color difference domain in the prior art, in the embodiment of the present invention, the color of the pixels in the inner circle and the outer circle is converted into the same color as the center pixel, effectively retaining the center pixel. Point the original pixel information of the surrounding pixels, so that each pixel in the image to be detected can be detected and corrected according to the original pixel information of the surrounding pixels, which improves the precision and accuracy of the detection and correction of the dead pixels.

[0095] Optionally, in step S201, after the electronic device establishes an N*N pixel matrix with the f-th pixel in the image to be detected as the center pixel, step S204 is performed.

[0096] S204: The electronic device determines whether the central pixel is an edge pixel.

[0097] Specifically, A1, the electronic device determines the variance value of the pixel values ​​of all the green G pixel points in the N*N pixel matrix.

[0098] A2. When the variance value is greater than the first threshold TH1, the electronic device confirms that the center pixel is an edge pixel; when the variance value is less than or equal to the first threshold TH1, the electronic device confirms The central pixel point is a non-edge pixel point.

[0099] Optionally, after the electronic device determines that the center pixel is an edge pixel, in step S202, the electronic device converts the color of each pixel of the inner circle and the outer circle into the color of the center pixel. , And before separately determining the converted pixel value of each pixel point, step S205 is executed.

[0100] S205: The electronic device performs same-color pre-correction on the N*N pixel matrix.

[0101] Specifically, when the electronic device determines that the difference between the pixel value of the first pixel and the pixel value of the second pixel is greater than the second threshold TH2, the electronic device modifies the pixel value of the first pixel to the first The pixel value of two pixels; the first pixel is the pixel in the outer circle that has the same color as the central pixel and has the largest pixel value; the second pixel is the pixel in the outer circle and the The central pixel has the same color and the next highest pixel value; or, when the electronic device determines that the difference between the pixel value of the third pixel and the pixel value of the fourth pixel is greater than the third threshold TH3, The pixel value of the third pixel is modified to the pixel value of the fourth pixel; the third pixel is the pixel in the outer circle with the same color as the central pixel and the smallest pixel value; The fourth pixel is a pixel in the outer circle that has the same color as the central pixel and has the next smallest pixel value.

[0102] In order to avoid the presence of dead pixels in the inner and outer rings and bring them into the interpolation calculation, the pixel values ​​of the same color pixels in the inner and outer rings are pre-judged, that is, all the same color pixels in the inner and outer rings If the difference between the maximum value and the second maximum value is greater than TH2, the second maximum value is used to replace the maximum value. If the difference between the second minimum value and the minimum value is greater than TH3, the minimum value is replaced by the second minimum value. Among them, TH2 and TH3 can have the same value.

[0103] In a possible implementation, in step S202, the electronic device converts the color of each pixel of the inner circle and the outer circle into the color of the central pixel, and determines the conversion of each pixel. The subsequent pixel value can be achieved in the following way:

[0104] B1, the electronic device separately determines the U color difference value and the V color difference value of each pixel;

[0105] Specifically, to image 3 The 7*7 pixel matrix shown as an example, for each point in the central 5x5 sub-block of the 7x7 pixel matrix, determine the red (English: Red, abbreviation: R) and green (English: Green, abbreviation: G), blue (English: Blue, abbreviation: B) three components, and then determine the U color difference value and V color difference value of each pixel, where the U color difference value and V color difference value meet the following formulas respectively :

[0106] U=B-G;

[0107] V=R-G;

[0108] Among them, when the electronic device determines the R, G, and B components of each pixel, it can use the same color channel mean interpolation method. Of course, other methods can also be used to determine the R, G, and B components of each pixel. The embodiments of the invention are not specifically limited here. To image 3 Take the G12 pixel in as an example. The R component and B component determined by the G12 pixel based on the same color channel mean interpolation method respectively satisfy the following formulas:

[0109] R=(R11+R13)/2;

[0110] B=(B02+B22)/2;

[0111] B2. The electronic device determines the average U color difference value Uavg based on the U color difference value of each pixel, and determines the average V color difference value Vavg based on the V color difference value of each pixel;

[0112] B3. For each heterochromatic pixel in the inner circle and the outer circle, the electronic device is based on the pixel value of each heterochromatic pixel, the U color difference value average, and the V color The average value of the difference determines the pixel value of each pixel of the different color after conversion.

[0113] To image 3 The 7x7 pixel matrix shown as an example, the converted pixel value of each different color pixel in the central 5x5 sub-block of the 7x7 pixel matrix satisfies the following formulas, and the converted central 5x5 sub-block is as Figure 5 Shown:

[0114] r12=G12+Vavg;

[0115] r14=G14+Vavg;

[0116] r21=G21+Vavg;

[0117] r22=B22-Uavg+Vavg;

[0118] r23=G23+Vavg;

[0119] r24=B24-Uavg+Vavg;

[0120] r25=G25+Vavg;

[0121] r32=G32+Vavg;

[0122] r34=G34+Vavg;

[0123] r41=G41+Vavg;

[0124] r42=B42-Uavg+Vavg;

[0125] r43=G43+Vavg;

[0126] r44=B44-Uavg+Vavg;

[0127] r45=G45+Vavg;

[0128] r52=G52+Vavg;

[0129] r54=G54+Vavg;

[0130] Using the color difference mean interpolation method can not only improve the interpolation accuracy, but also effectively reduce the influence on the interpolation when there are dead pixels around, and improve the accuracy of the dead pixel detection.

[0131] Optionally, in step S202, the electronic device converts the color of each pixel of the inner circle and the outer circle into the color of the center pixel, and determines the converted pixel value of each pixel. After that, in step S203, the electronic device determines that the central pixel is a dead pixel based on the converted pixel value of each pixel, and performs step S206 before correcting the central pixel.

[0132] S206: The electronic device performs inner circle and outer circle pre-correction on the N*N pixel matrix.

[0133] Specifically, the electronic device determines that the maximum pixel value after the inner circle conversion is greater than the second-largest pixel value after the global conversion, and the maximum pixel value after the inner circle conversion is the second-largest pixel value after the global conversion. When the difference between the values ​​is greater than the fourth threshold TH4, modify the maximum pixel value after the inner circle conversion to the second largest pixel value after the global conversion; the global includes the inner circle and the outer circle; or,

[0134] The electronic device determines that the minimum pixel value after the inner circle conversion is smaller than the second-smallest pixel value after the global conversion, and the minimum pixel value after the inner circle conversion is the second-smallest pixel value after the global conversion When the difference between the two is greater than the fifth threshold TH5, the smallest pixel value after the inner circle conversion is modified to the second smallest pixel value after the global conversion; or,

[0135] The electronic device determines that the second-largest pixel value after conversion of the inner circle is greater than the third-largest pixel value after the global conversion, and the second-largest pixel value after the inner circle conversion is the same as that after the global conversion. When the difference between the third largest pixel value is greater than the sixth threshold TH6, modify the second largest pixel value after the inner circle conversion to the third largest pixel value after the global conversion; or,

[0136] The electronic device determines that the converted second-smallest pixel value of the inner circle is less than the third-smallest pixel value after the global conversion, and the second-smallest pixel value after the conversion of the inner circle is the same as the second-smallest pixel value after the global conversion. When the difference between the third smallest pixel values ​​is greater than the seventh threshold TH7, the second smallest pixel value after the inner circle conversion is modified to the third smallest pixel value after the global conversion.

[0137] Among them, TH4, TH5, TH6, and TH7 can be determined based on the noise estimation value of the N*N pixel matrix.

[0138] In a possible implementation, in step S203, the electronic device determines that the central pixel is a dead pixel based on the converted pixel value of each pixel, and corrects the central pixel, which can be performed by Realize as follows:

[0139] C1. The electronic device determines the maximum value among the converted pixel values ​​of each pixel in the inner circle and the outer circle, and determines that the pixel value of the central pixel is greater than the maximum value, and the When the difference between the pixel value of the central pixel and the maximum value is greater than the eighth threshold TH8, it is determined that the central pixel is a dead pixel.

[0140] Wherein, TH8 can be determined based on the noise estimation value of the N*N pixel matrix.

[0141] C2. The electronic device determines at least one pixel pair, and respectively determines the pixel pair with the smallest pixel value difference in the at least one pixel pair; the pixel pair is formed by the same color as the central pixel in the outer circle , And the central pixel is a symmetrical point composed of two symmetrical pixel values;

[0142] Specifically, the electronic device determines that two pixels of the same color with the center pixel in the 0° direction are a pixel pair, two pixels with the same color in the center pixel in the 45° direction are a pixel pair, and the center pixel is in the 90° direction. Two pixels of the same color are a pixel pair, and two pixels of the same color with the center pixel in the 135° direction are calculated as a pixel pair, see Image 6 Shown.

[0143] participate Image 6 Take the pixel pair shown as an example, grad0, grad45, grad90, and grad135 of R33 satisfy the following formulas:

[0144] grad0=|R31-R35|;

[0145] grad45=|R51-R15|;

[0146] grad90=|R13-R53|;

[0147] grad135=|R11-R55|;

[0148] C3. The electronic device corrects the pixel value of the central pixel point to the larger value of the pixel pair with the smallest pixel value difference.

[0149] In another possible implementation manner, in step S203, the electronic device determines that the central pixel is a dead pixel based on the converted pixel value of each pixel, corrects the central pixel, and further It can be achieved in the following ways:

[0150] D1. The electronic device determines the minimum value of the converted pixel values ​​of each pixel in the inner circle and the outer circle, and determines that the pixel value of the center pixel is less than the minimum value, and the When the difference between the pixel value of the central pixel and the minimum value is greater than the ninth threshold TH9, determining that the central pixel is a bad pixel;

[0151] Wherein, TH9 can be determined based on the noise estimation value of the N*N pixel matrix.

[0152] D2. The electronic device determines at least one pixel pair, and respectively determines the pixel pair with the smallest pixel value difference in the at least one pixel pair; the pixel pair has the same color as the central pixel in the outer circle , And the central pixel point is composed of the pixel values ​​of two symmetrical pixels;

[0153] D3. The electronic device corrects the pixel value of the center pixel to the smaller value of the pixel pair with the smallest pixel value difference.

[0154] Optionally, after the electronic device determines that the center pixel is a non-edge pixel, step S207 is executed.

[0155] S207: The electronic device performs outer circle pre-correction on the N*N pixel matrix.

[0156] Specifically, if the electronic device determines that the difference between the largest pixel value and the second largest pixel value in the same color pixels in the outer circle is greater than the tenth threshold TH10, the second largest pixel value is substituted for the largest pixel value;

[0157] If the electronic device determines that the difference between the smallest pixel value and the second smallest pixel value in the same color pixel points of the outer circle is greater than the eleventh threshold TH11, the smallest pixel value is replaced by the second smallest pixel value.

[0158] If the electronic device determines that the difference between the second-largest pixel value and the third-largest pixel value in the pixels of the same color in the outer circle is greater than the twelfth threshold TH12, the third-largest pixel value replaces the largest image value;

[0159] If the electronic device determines that the difference between the second-smallest pixel value and the third-smallest pixel value in the same-color pixel points of the outer circle is greater than the thirteenth threshold TH13, the third-smallest pixel value is substituted for the smallest pixel value.

[0160] Among them, TH10, TH11, TH12, and TH13 can be determined based on the noise estimation value of the N*N pixel matrix.

[0161] Optionally, after step S207 is performed, step S208 is performed.

[0162] S208: The electronic device determines that the central pixel is a dead pixel based on the same color pixels of the outer circle, and corrects the central pixel.

[0163] Specifically, when the electronic device determines the maximum value among the pixel values ​​of pixels of the same color in the outer circle, and determines that the pixel value of the center pixel is greater than the maximum value, and the pixel of the center pixel When the difference between the value and the maximum value is greater than the fourteenth threshold TH14, it is determined that the central pixel is a dead pixel. The electronic device determines at least one pixel pair, and respectively determines the pixel pair with the smallest pixel value difference in the at least one pixel pair; the electronic device corrects the pixel value of the center pixel to the pixel value difference The larger value of the smallest pixel pair.

[0164] Wherein, the pixel pair is composed of the pixel values ​​of two pixels in the outer circle that have the same color as the central pixel and are symmetrical with the central pixel. TH14 can be determined based on the noise estimation value of the N*N pixel matrix.

[0165] When the electronic device determines the minimum value of the converted pixel values ​​of each pixel in the inner circle and the outer circle, and determines that the pixel value of the center pixel is less than the minimum value, and the center When the difference between the pixel value of the pixel and the minimum value is greater than the fifteenth threshold TH15, it is determined that the central pixel is a dead pixel; the electronic device determines at least one pixel pair, and determines the at least one pixel respectively A pixel pair with the smallest pixel value difference in the center; the electronic device corrects the pixel value of the central pixel point to the smaller value of the pixel pair with the smallest pixel value difference.

[0166] Wherein, the pixel pair is composed of the pixel values ​​of two pixels in the outer circle that have the same color as the central pixel and are symmetrical with the central pixel. TH15 can be determined based on the noise estimation value of the N*N pixel matrix.

[0167] In the embodiment of the present invention, for the image to be detected, the electronic device establishes an N*N pixel matrix with the f-th pixel in the image to be detected as the central pixel; the electronic device divides each pixel of the inner circle and the outer circle The color of the dot is converted into the color of the central pixel, and the converted pixel value of each pixel is respectively determined; the electronic device determines the central pixel based on the converted pixel value of each pixel The dot is a bad dot, and the center pixel dot is corrected. Compared with the method of detecting and correcting the color difference domain in the prior art, in the embodiment of the present invention, the color of the pixels in the inner circle and the outer circle is converted into the same color as the center pixel, effectively retaining the center pixel. Point the original pixel information of the surrounding pixels, so that each pixel in the image to be detected can be detected and corrected according to the original pixel information of the surrounding pixels, which improves the precision and accuracy of the detection and correction of the dead pixels.

[0168] Based on and figure 2 Corresponding to the same inventive concept of the method embodiment, the embodiment of the present invention provides an image defect correction device 70, the structure of the device is as Figure 7 As shown, it includes a creation module 71, a conversion module 72 and a correction module 73, where:

[0169] The creation module 71 is configured to establish an N*N pixel matrix with the f-th pixel in the image to be detected as the central pixel for the image to be detected; the f is not greater than the number of passes included in the image to be detected A positive integer of the total number of pixels; the N is an odd number greater than 1;

[0170] The conversion module 72 is configured to convert the color of each pixel of the inner circle and the outer circle of the N*N pixel matrix established by the creation module 71 into the color of the central pixel, and respectively determine the The converted pixel value of each pixel; the inner circle is a pixel in the N*N pixel matrix that is adjacent to the central pixel; the outer circle is in the N*N pixel matrix B pixels separated by one row and/or one column apart from the central pixel; the a and b are both integers greater than 1;

[0171] The correction module 73 is configured to determine that the central pixel is a dead pixel based on the converted pixel value of each pixel determined by the conversion module 72, and correct the central pixel.

[0172] Optionally, the conversion module 72 is specifically configured to: respectively determine the U color difference value and the V color difference value of each pixel; and determine the U color difference value based on the U color difference value of each pixel The average value of the V color difference value is determined based on the V color difference value of each pixel; for each different color pixel in the inner circle and the outer ring, based on each different color pixel The pixel value of the pixel value, the average value of the U color difference value, and the average value of the V color difference value determine the pixel value after the conversion of each different color pixel; the different color pixel is the pixel value in the image to be detected and the Pixels with different colors in the center pixel.

[0173] Optionally, the correction module 73 is specifically configured to: determine the maximum value of the converted pixel values ​​of each pixel in the inner circle and the outer circle, and determine that the pixel value of the center pixel is greater than When the maximum value and the difference between the pixel value of the central pixel and the maximum value is greater than the first threshold, the central pixel is determined to be a dead pixel; at least one pixel pair is determined, and the At least one pixel pair has the smallest pixel value difference; the pixel pair consists of two symmetrical pixels in the outer circle that have the same color as the central pixel and are symmetrical with the central pixel The pixel value of the center pixel is corrected to the larger value of the pixel pair with the smallest pixel value difference.

[0174] Optionally, the correction module 73 is specifically configured to: determine the minimum value of the converted pixel values ​​of each pixel in the inner circle and the outer circle, and determine that the pixel value of the center pixel is less than When the minimum value and the difference between the pixel value of the central pixel point and the minimum value is greater than a second threshold, the central pixel point is determined to be a dead pixel; at least one pixel pair is determined, and the At least one pixel pair has the smallest pixel value difference; the pixel pair consists of two symmetrical pixels in the outer circle that have the same color as the central pixel and are symmetrical with the central pixel The pixel value of the center pixel is corrected to the smaller value of the pixel pair with the smallest pixel value difference.

[0175] Optionally, the device further includes a first pre-correction module 74; the first pre-correction module 74 is configured to:

[0176] Before the conversion module 72 converts the color of each pixel of the inner circle and the outer circle into the color of the center pixel, and determines the converted pixel value of each pixel, it determines the first When the difference between the pixel value of the pixel and the pixel value of the second pixel is greater than the third threshold, the pixel value of the first pixel is modified to the pixel value of the second pixel; the first pixel A point is a pixel in the outer circle with the same color as the central pixel and with the largest pixel value; the second pixel is a pixel in the outer circle with the same color as the central pixel and the second largest pixel value Point; or,

[0177] Before the conversion module 72 converts the color of each pixel of the inner circle and the outer circle into the color of the center pixel, and determines the converted pixel value of each pixel, before determining the third When the difference between the pixel value of the pixel and the pixel value of the fourth pixel is greater than the fourth threshold, the pixel value of the third pixel is modified to the pixel value of the fourth pixel; the third pixel A dot is a pixel in the outer circle that has the same color as the central pixel and has the smallest pixel value; the fourth pixel is a pixel in the outer circle that has the same color as the central pixel and has the next smallest pixel value point.

[0178] Optionally, the device further includes a second pre-correction module 75; the second pre-correction module 75 is configured to:

[0179] After the conversion module 72 converts the color of each pixel of the inner circle and the outer circle into the color of the center pixel, and determines the converted pixel value of each pixel, after determining the When the maximum pixel value after the inner circle conversion is greater than the second-largest pixel value after the global conversion, and the difference between the maximum pixel value after the inner circle conversion and the second-largest pixel value after the global conversion is greater than the fifth threshold, Modify the maximum pixel value after conversion of the inner circle to the second largest pixel value after the global conversion; the global includes the inner circle and the outer circle; or,

[0180] After the conversion module 72 converts the color of each pixel of the inner circle and the outer circle into the color of the center pixel, and determines the converted pixel value of each pixel, after determining the The smallest pixel value after the inner circle conversion is smaller than the second smallest pixel value after the global conversion, and the difference between the smallest pixel value after the inner circle conversion and the second smallest pixel value after the global conversion is greater than a sixth threshold , Modify the smallest pixel value after the inner circle conversion to the second smallest pixel value after the global conversion; or,

[0181] After the conversion module 72 converts the color of each pixel of the inner circle and the outer circle into the color of the center pixel, and determines the converted pixel value of each pixel, after determining the The second largest pixel value after the inner circle conversion is greater than the third largest pixel value after the global conversion, and the difference between the second largest pixel value after the inner circle conversion and the third largest pixel value after the global conversion is When the difference is greater than the seventh threshold, modify the second largest pixel value after the inner circle conversion to the third largest pixel value after the global conversion; or

[0182] After the conversion module 72 converts the color of each pixel of the inner circle and the outer circle into the color of the center pixel, and determines the converted pixel value of each pixel, after determining the The second smallest pixel value after the inner circle conversion is smaller than the third smallest pixel value after the global conversion, and the difference between the second smallest pixel value after the inner circle conversion and the third smallest pixel value after the global conversion When the difference is greater than the eighth threshold, the second smallest pixel value after the inner circle conversion is modified to the third smallest pixel value after the global conversion.

[0183] Optionally, the device further includes: a determining module 76, configured to convert the color of each pixel of the inner circle and the outer circle into the color of the center pixel in the conversion module 72, and determine the color of each pixel. Before the converted pixel value of each pixel point, determine the variance value of the pixel values ​​of all green G pixel points in the N*N pixel matrix, and determine that the variance value is greater than a ninth threshold.

[0184] Optionally, after determining the variance value of the pixel values ​​of all the green G pixels in the N*N pixel matrix, the determining module 76 is further configured to: determine that the variance value is less than or equal to the ninth Threshold

[0185] The correction module 73 is further configured to: determine that the central pixel is a dead pixel based on the pixel value of each pixel of the same color in the outer ring, and correct the central pixel, and the pixel of the same color is A pixel in the image to be detected that has the same color as the central pixel.

[0186] Optionally, the device further includes: a mirroring module 77, configured to set the N*N pixel matrix before the creation module 71 takes the fth pixel in the image to be detected as the center pixel The image to be detected is mirror-flanged for the upper boundary, lower boundary, left boundary, and right boundary.

[0187] Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0188] The present invention is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated for use In the process Figure one Process or multiple processes and/or boxes Figure one A device with functions specified in a block or multiple blocks.

[0189] These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The device is implemented in the process Figure one Process or multiple processes and/or boxes Figure one Functions specified in a box or multiple boxes.

[0190] These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment. Instructions are provided to implement the process Figure one Process or multiple processes and/or boxes Figure one Steps of functions specified in a box or multiple boxes.

[0191] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0192] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. In this way, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.