Encoding apparatus and computer program

Abstract

The objective of this invention is to improve the compression efficiency of image data. An encoding apparatus is provided for encoding image data from a camera sensor having a three-primary-color filter and an infrared filter in the visible light wavelength region. The encoding apparatus includes an encoding processing unit that encodes visible light image frames and infrared image frames after converting their color space from RGB to YUV. The encoding processing unit is configured to, when predicting the pixel values ​​of the encoding target blocks of the visible light image frames and infrared image frames, select a prediction mode from intra-frame prediction, inter-frame prediction, and inter-viewpoint prediction; and encode the difference between the predicted pixel values ​​obtained by predicting the pixel values ​​of the encoding target blocks of the visible light image frames and infrared image frames using the prediction mode and the actual pixel values ​​of the encoding target blocks of the color-converted visible light image frames and infrared image frames.

Description

Technical Field

[0001] This invention relates to an encoding device and a computer program. Background Technology

[0002] In Patent Document 1, as a conventional image encoding device for encoding images obtained by a camera sensor with R, G, B three primary color filters and an infrared filter having visible light wavelength regions, a device is disclosed that is configured to generate differential data representing the difference between image data of any one of the three primary color filters determined according to the intensity distribution of the transmission wavelength of the infrared filter and image data obtained through the infrared filter, and to encode the image data and differential data of each of the three primary color filters.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2023-107583 Summary of the Invention

[0004] However, the aforementioned conventional image encoding devices have the following problems: because they do not encode the image data (moving image data) in a way that takes into account the temporal redundancy of the image data or the correlation between the brightness components of the visible light image and the pixel values ​​of the infrared image, they cannot achieve sufficient data compression.

[0005] This invention was developed with this problem in mind, and its purpose is to improve the compression efficiency of image data.

[0006] To address the aforementioned issues, an encoding device according to one aspect of the present invention encodes image data from a camera sensor having a three-primary-color filter and an infrared filter in the visible light wavelength region. The encoding device comprises:

[0007] The system comprises a developing unit that develops image data to generate visible light image frames and infrared image frames; a color space conversion unit that converts the color space of the visible light image frames and infrared image frames from RGB color space to YUV color space; and an encoding unit that encodes the color-converted visible light image frames and infrared image frames. The encoding unit is configured to select a prediction mode from intra-frame prediction, inter-frame prediction, and inter-viewpoint prediction when predicting the pixel values ​​of the encoding target blocks of the visible light image frames and infrared image frames; and to encode the difference between the predicted pixel values ​​obtained by predicting the pixel values ​​of the encoding target blocks of the visible light image frames and infrared image frames using the prediction mode and the actual pixel values ​​of the encoding target blocks of the color-converted visible light image frames and infrared image frames.

[0008] Furthermore, a computer program according to one aspect of the present invention causes a computer to perform the following processing: developing image data from a camera sensor having a 3-primary-color filter and an infrared filter in the visible light wavelength region to generate visible light image frames and infrared image frames; converting the color space of the visible light image frames and infrared image frames from RGB color space to YUV color space; when predicting the pixel values ​​of the encoded object blocks of the visible light image frames and infrared image frames, selecting a prediction mode from intra-frame prediction, inter-frame prediction, and inter-viewpoint prediction; and encoding the difference between the predicted pixel values ​​obtained by predicting the pixel values ​​of the encoded object blocks of the visible light image frames and infrared image frames using the prediction mode and the actual pixel values ​​of the encoded object blocks of the color-converted visible light image frames and infrared image frames.

[0009] Invention Effects

[0010] According to these methods of the present invention, image data is encoded by inter-frame prediction that takes into account the temporal redundancy of image data or by inter-viewpoint prediction that takes into account the correlation between the brightness component of the visible light image and the pixel value of the infrared image, thereby improving the compression efficiency of image data. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the image encoding device.

[0012] Figure 2 This is a flowchart illustrating an example of the prediction mode setting process.

[0013] Figure 3 This is a flowchart illustrating an example of the predicted value calculation process.

[0014] Figure 4 This is a flowchart illustrating an example of the reference viewpoint selection process. Detailed Implementation

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Furthermore, in the following description, the same reference numerals will be used to denote the same constituent elements.

[0016] Figure 1 This is a schematic structural diagram of the encoding device 100 according to the first embodiment of the present invention.

[0017] The encoding device 100 includes an imaging unit 10, a developing unit 20, a color space conversion unit 30, and an encoding processing unit 40. Furthermore, each process in the encoding device 100 is performed by a processor having one or more central processing units (CPUs) and their peripheral circuitry executing programs stored in a storage device.

[0018] The camera unit 10 is equipped with a camera sensor that has a three-primary-color filter (R, G, B) with visible light wavelengths and an infrared filter, and inputs image data (e.g., Bayer RAW data with RGB-IR pixel arrangement) output from the camera sensor to the developing unit 20.

[0019] The developing unit 20 performs various processing (e.g., de-mosaic processing) on ​​the image data output from the imaging unit 10 to generate an image frame, and outputs the image frame into two image frames with different visual characteristics, a visible light image frame and an infrared image frame, and inputs them into the color space conversion unit 30.

[0020] The color space conversion unit 30 converts the color spaces of the visible light image frame and the infrared image frame output from the developing unit from the RGB color space to the YUV color space, which is composed of a luminance component (Y) and a pigment component (U, V), and outputs the converted color space to the encoding processing unit 40. In this embodiment, the color space conversion unit 30 converts the RGB pixel values ​​of the visible light image frame into pixel values ​​of the YUV color space, which is composed of a luminance component (Y) and a pigment component (U, V). For the infrared image frame, the IR pixel values ​​are converted into pixel values ​​of the YUV color space, where the luminance component (Y) and the pigment component (U, V) are set to zero.

[0021] The encoding processing unit 40 includes an encoding control unit 41, a prediction value calculation unit 42, and a prediction difference encoding unit 43, and encodes visible light image frames and infrared image frames output from the color space conversion unit 30.

[0022] The encoding control unit 41 includes a reference viewpoint selection unit 411 and a prediction mode setting unit 412.

[0023] The reference viewpoint selection unit 411 selects either a visible light image sequence consisting of multiple temporally consecutive visible light image frames or an infrared image sequence consisting of multiple temporally consecutive infrared image frames as the reference viewpoint. In this embodiment, the reference viewpoint selection unit 411 selects all visible light image sequences with more information during the daytime as the reference viewpoint.

[0024] The prediction mode setting unit 412 sets the prediction mode of the visible light image sequence and the infrared image sequence frame by frame from intra prediction, inter prediction and inter-view prediction until the encoding of the object sequence is completed.

[0025] Figure 2 This is a flowchart illustrating an example of the prediction mode setting process performed in the prediction mode setting unit 412.

[0026] In step S1, the prediction mode setting unit 412 sequentially selects image frames (hereinafter referred to as "prediction mode setting frames") from the visible light image sequence and the infrared image sequence to set the prediction mode.

[0027] In step S2, the prediction mode setting unit 412 determines whether the prediction mode setting frame selected in step S1 is the opening frame of a Group of Pictures (GOP) that serves as the starting point for special playback such as fast forward or rewind. If the prediction mode setting frame is the opening frame of a GOP, the prediction mode setting unit 412 proceeds to step S3. On the other hand, if the prediction mode setting frame is not the opening frame of a GOP, the prediction mode setting unit 412 proceeds to step S6.

[0028] In step S3, the prediction mode setting unit 412 determines whether the prediction mode setting frame is an image frame of a sequence selected as the reference viewpoint. Therefore, in this embodiment, if the prediction mode setting frame is an image frame of a visible light image sequence, the prediction mode setting unit 412 proceeds to step S4, and if it is an image frame of an infrared image sequence, it proceeds to step S5.

[0029] In step S4, the prediction mode setting unit 412 sets the prediction mode of the visible light image frame, which serves as the prediction mode setting frame, to intra-frame prediction.

[0030] In step S5, the prediction mode setting unit 412 sets the prediction mode of the infrared image frame, which serves as the prediction mode setting frame, to inter-viewpoint prediction.

[0031] In step S6, the prediction mode setting unit 412 determines whether the prediction mode setting frame is an image frame of a sequence selected as the reference viewpoint. Therefore, in this embodiment, if the prediction mode setting frame is an image frame of a visible light image sequence, the prediction mode setting unit 412 proceeds to step S7, and if it is an image frame of an infrared image sequence, it proceeds to step S8.

[0032] In step S7, the prediction mode setting unit 412 sets the prediction mode of the visible light image frame, which serves as the prediction mode setting frame, to inter-frame prediction.

[0033] In step S8, the prediction mode setting unit 412 sets the prediction mode of the infrared image frame, which serves as the prediction mode setting frame, to inter-viewpoint prediction.

[0034] Thus, regarding the first frame of a GOP, the prediction mode setting unit 412 sets the prediction mode to intra-frame prediction if the frame is part of a sequence of image frames selected as the reference viewpoint, and sets the prediction mode to inter-viewpoint prediction if the frame is part of a sequence of image frames not selected as the reference viewpoint. Then, regarding frames other than the first frame of a GOP, the prediction mode setting unit 412 sets the prediction mode to inter-frame prediction if the frame is part of a sequence of image frames selected as the reference viewpoint, and sets the prediction mode to inter-viewpoint prediction if the frame is part of a sequence of image frames not selected as the reference viewpoint.

[0035] Therefore, in this embodiment, since the visible light image sequence is uniformly selected as the reference viewpoint, the prediction mode is set to intra-frame prediction for visible light image frames starting with a GOP, and to inter-frame prediction for visible light image frames other than those starting with a GOP. Then, for infrared image frames, regardless of whether they are starting with a GOP, the prediction mode is set to inter-viewpoint prediction.

[0036] Return to Figure 1 The prediction value calculation unit 42 calculates the predicted value (hereinafter referred to as "predicted pixel value") of the pixel value of the encoded object block of the prediction mode setting frame according to the prediction mode of the prediction mode setting frame set in the prediction mode setting unit 412, and inputs it to the difference calculation unit.

[0037] Figure 3 This is a flowchart illustrating an example of the prediction value calculation process performed in the prediction value calculation unit 42.

[0038] In step S21, the prediction value calculation unit 42 determines whether the prediction mode of the prediction mode setting frame is intra-frame prediction. If the prediction mode of the prediction mode setting frame is intra-frame prediction, the prediction value calculation unit 42 proceeds to step S22. On the other hand, if the prediction mode of the prediction mode setting frame is not intra-frame prediction, the prediction value calculation unit 42 proceeds to step S23.

[0039] In step S22, the budget value calculation unit predicts the pixel values ​​of the encoded object block of the visible light image frame, which is the prediction mode setting frame, by intra-frame prediction.

[0040] Intra-frame prediction is a method for predicting the pixel value of a coded object block by referring to the pixel values ​​of adjacent decoded blocks within the same frame. In intra-frame prediction, the predicted pixel value of the coded object block is calculated based on the pixel values ​​of adjacent blocks (decoded blocks) of the coded object block decoded by the local decoding unit 435 (described later).

[0041] In step S23, the prediction value calculation unit 42 determines whether the prediction mode of the prediction mode setting frame is inter-frame prediction. If the prediction mode of the prediction mode setting frame is inter-frame prediction, the prediction value calculation unit 42 proceeds to step S24. On the other hand, if the prediction mode of the prediction mode setting frame is not inter-frame prediction, the prediction value calculation unit 42 proceeds to step S25.

[0042] In step S24, the budget value calculation unit predicts the pixel values ​​of the encoded object block of the visible light image frame, which is the prediction mode setting frame, by inter-frame prediction.

[0043] Inter-frame prediction is a method for predicting the pixel values ​​of coded object blocks in the current image frame using information from past or future image frames. In inter-frame prediction, firstly, an image frame that is temporally close to the current image frame (e.g., the immediately following image frame) is selected from the image frames stored in the frame memory 436 (described later) as a reference frame. Then, an action search is performed to determine which corresponding block in the current image frame corresponds to the coded object block in the reference frame. The positional difference between the corresponding block in the current image frame and the corresponding block in the reference frame is calculated as an action vector value, and the pixel value of the corresponding block in the reference frame is calculated as the predicted pixel value of the coded object block.

[0044] In step S25, the prediction value calculation unit 42 predicts the pixel values ​​of the encoded object block of the infrared image frame, which is the prediction mode setting frame, by inter-viewpoint prediction.

[0045] In inter-viewpoint prediction, firstly, a visible light image frame that serves as the reference viewpoint is selected from the image frames stored in the frame memory 436 (described later) as a reference frame. Then, the encoded object blocks of the reference viewpoint image frame and the visually correlated infrared image frame are searched to determine which region within the selected visible light image frame corresponds to them. The line-of-sight vector value representing the positional deviation between viewpoints is calculated, and the predicted pixel value of the encoded object block of the infrared image frame, which serves as the prediction mode setting frame, is calculated as the pixel value of the corresponding block within the selected visible light image frame.

[0046] At this point, it is assumed that the pixel values ​​of the infrared image frame are highly correlated with the luminance component values ​​(Y) of the visible light image frame. Therefore, regarding the luminance component value (Y), the luminance component value (Y) of the visible light image frame selected as the reference frame can be set as the predicted pixel value of the luminance component value (Y) of the encoded object block of the infrared image frame, which is used as the prediction mode setting frame. On the other hand, the predicted pixel value of the pigment component values ​​(U, V) of the encoded object block of the infrared image frame can be set to zero. This allows for maintaining the amount of data related to the visually important luminance component to suppress image quality degradation caused by data compression, and reducing the amount of data related to the visually unimportant pigment component to improve the compression efficiency of the image data.

[0047] Furthermore, when the predicted pixel value of the pigment component values ​​(U, V) of the encoded object block of the infrared image frame is set to zero, for example, a weighting coefficient multiplied by the pigment component values ​​(U, V) of the visible light image frame can be set so that the predicted pixel value of the pigment component values ​​(U, V) of the predicted object block of the infrared image frame, which is the prediction mode setting frame, becomes zero.

[0048] Furthermore, in inter-viewpoint prediction, the disparity vector value can be set to zero to search for which region within the visible light image frame selected as the reference frame corresponds to the encoding object block of the reference viewpoint image frame and the corresponding block of the corresponding infrared image frame. Thus, by pre-setting the disparity vector value to zero to search for the correspondence between the encoding object block of the infrared image frame and the corresponding block of the visible light image frame, the processing load of the search process can be reduced, thereby reducing power consumption. Moreover, since the disparity vector value is zero, the amount of data (symbols) generated when encoding residual data containing the disparity vector value can be reduced. This is particularly useful when encoding two images output from a camera that captures both visible light and infrared images from a single sensor.

[0049] Return to Figure 1 The prediction difference coding unit 43 includes a difference calculation unit 431, a conversion / quantization unit 432, an entropy coding unit 433, an inverse conversion / inverse quantization unit 434, a local decoding unit 435, and a frame memory 436.

[0050] The difference calculation unit 431 calculates the difference (residual) between the predicted pixel value of the encoded object block of the prediction mode setting frame output from the prediction value calculation unit 42 and the actual pixel value of the prediction mode setting frame output from the color space conversion unit 30 to generate residual data, which is then input to the conversion / quantization unit 432. When the prediction value calculation unit 42 calculates a motion vector value or a disparity vector value, the residual data includes the motion vector value or the disparity vector value.

[0051] The conversion / quantization unit 432 converts the residual data output from the difference calculation unit 431 into frequency components to reduce unwanted information, and quantizes the frequency-converted residual data to reduce the data size before inputting it into the entropy encoding unit 433 and the inverse conversion / inverse quantization unit 434. Furthermore, regarding the quantization of the residual data of the pigment components in the infrared image frame, the quantization parameter can be set to the maximum value, or it can be truncated to zero.

[0052] The entropy encoding unit 433 compresses the quantized residual data output from the conversion / quantization unit 432 through entropy encoding.

[0053] The inverse conversion / inverse quantization unit 434 decodes and outputs the residual data of the encoded object block of the prediction mode setting frame output from the conversion / quantization unit 432, and inputs it to the local decoding unit 435.

[0054] The local decoding unit 435 adds the difference between the composite residual data and the predicted pixel value of the predicted object block of the prediction mode setting frame, and stores it in the frame memory 436 as the pixel value of the decoded block of the prediction mode setting frame.

[0055] Frame memory 436 temporarily stores the pixel values ​​of the decoded blocks of the prediction mode setting frame used in intra-frame prediction or the image frames used in inter-frame prediction and inter-viewpoint prediction.

[0056] The encoding apparatus 100 of this embodiment described above encodes image data from a camera sensor having a three-primary-color filter and an infrared filter in the visible light wavelength region. The encoding apparatus 100 includes: a developing unit 20 that develops the image data to generate visible light image frames and infrared image frames; a color space conversion unit 30 that converts the color space of the visible light image frames and infrared image frames from RGB color space to YUV color space; and an encoding processing unit 40 that encodes the color-converted visible light image frames and infrared image frames.

[0057] When predicting the pixel values ​​of the coded object blocks of the visible light image frame and the infrared image frame, the encoding processing unit 40 selects a prediction mode from intra-frame prediction, inter-frame prediction, and inter-viewpoint prediction; and encodes the difference between the predicted pixel values ​​obtained by predicting the pixel values ​​of the coded object blocks of the visible light image frame and the infrared image frame after color conversion and the actual pixel values ​​of the coded object blocks of the visible light image frame and the infrared image frame.

[0058] Therefore, by encoding image data through inter-frame prediction that considers the temporal redundancy of image data or through inter-viewpoint prediction that considers the correlation between the luminance components of visible light image frames and the pixel values ​​of infrared image frames, the compression efficiency of image data can be improved. Furthermore, since the color spaces of visible light and infrared image frames can be converted from RGB to YUV color space before encoding, information on visually less important pigment components can be compressed, thereby further improving the compression efficiency of image data.

[0059] The encoding processing unit 40 is configured such that, when inter-viewpoint prediction is selected as the prediction mode, the disparity vector value is set to zero, a corresponding block that is related to the coded object block in another image frame not selected as a reference viewpoint is searched from the blocks in one image frame selected as a reference viewpoint in the visible light image frame and the infrared image frame, and the predicted pixel value of the coded object block in the other image frame is calculated based on the pixel value of the corresponding block in the image frame selected as the reference viewpoint.

[0060] Thus, when searching for a corresponding block in a block within an image frame selected as a reference viewpoint that is related to a coded object block in another image frame not selected as a reference viewpoint, the power consumption can be reduced by pre-setting the disparity vector value to zero. Furthermore, since the disparity vector value is zero, the amount of data (symbols) generated during encoding can be reduced.

[0061] Furthermore, the encoding processing unit 40 is configured such that, when inter-viewpoint prediction is selected as the prediction mode, it searches for a corresponding block that is related to the encoded object block in another image frame not selected as a reference viewpoint from the blocks in one image frame selected as a reference viewpoint in the visible light image frame and the infrared image frame, and sets the luminance component value of the corresponding block in the image frame selected as the reference viewpoint as the predicted pixel value of the luminance component of the encoded object block in the other image frame, and sets the predicted pixel value of the pigment component of the encoded object block in the other image frame to zero.

[0062] Therefore, it is possible to maintain the amount of data related to the luminance components with high visual importance to suppress image quality degradation caused by data compression, and reduce the amount of data related to the pigment components with low visual importance to improve the compression efficiency of image data.

[0063] (Second Implementation)

[0064] Next, the second embodiment of the present invention will be described. This embodiment differs from the first embodiment in that the image sequence used as the reference viewpoint is selected based on the photographic scene of the image data. The differences will be explained below.

[0065] In the first embodiment described above, the reference viewpoint selection unit 411 selects all visible light image sequences with more information during the day as the reference viewpoint. However, at night, visible light image frames become dark and noisy due to insufficient illumination, while infrared image frames are usually clear images.

[0066] Therefore, the reference viewpoint selection unit 411 in this embodiment selects an image sequence as the reference viewpoint based on the photographic scene of the image data. Specifically, during the day, a visible light image that provides a clearer image than an infrared image is selected as the reference viewpoint, and at night, an infrared image that provides a clearer image than a visible light image is selected as the reference viewpoint. Thus, a clear image can be used as the reference viewpoint to predict the pixel values ​​of the encoded object blocks of the image frame, thereby suppressing prediction errors.

[0067] Figure 4 This is a flowchart illustrating an example of the reference viewpoint selection process implemented in the reference viewpoint selection unit 411 of this embodiment.

[0068] In step S31, the reference viewpoint selection unit 411 determines whether the reference viewpoint selection mode is set to a scene-corresponding mode that selects the reference viewpoint based on the image scene, or whether it is set to a user-defined mode that sets a user-arbitrarily selected image sequence from the visual image sequence and the infrared image sequence as the reference viewpoint. If the reference viewpoint selection mode is set to the scene-corresponding mode, the reference viewpoint selection unit 411 proceeds to step S32. On the other hand, if the reference viewpoint selection mode is set to the user-defined mode, the reference viewpoint selection unit proceeds to step S35.

[0069] In step S32, the reference viewpoint selection unit 411 determines whether the shooting scene of the image data is nighttime. If the shooting scene of the image data is nighttime, the reference viewpoint selection unit 411 proceeds to step S33. On the other hand, if the shooting scene of the image data is not nighttime (i.e., it is daytime), the reference viewpoint selection unit 411 proceeds to step S34.

[0070] In step S33, the reference viewpoint selection unit 411 selects an infrared image sequence as the reference viewpoint.

[0071] In step S34, the reference viewpoint selection unit 411 selects a visible light image sequence as the reference viewpoint.

[0072] In step S35, the reference viewpoint selection unit 411 selects a user-preset image sequence from the visual image sequence and the infrared image sequence as the reference viewpoint.

[0073] The encoding processing unit 40, which constitutes part of the reference viewpoint selection unit 411 in this embodiment, is configured to select either a visible light image frame or an infrared image frame as the reference viewpoint based on the photography scene of the image data. If the photography scene is daytime, the visible light image frame is selected as the reference viewpoint; if the photography scene is nighttime, the infrared image frame is selected as the reference viewpoint.

[0074] Therefore, during the daytime, the visible light image, which provides a clearer image than the infrared image, is used as the reference viewpoint, and at night, the infrared image, which provides a clearer image than the visible light image, is used as the reference viewpoint to predict the pixel values ​​of the coded object blocks of the image frame, thus suppressing prediction errors.

[0075] The embodiments of the present invention have been described above. However, the above embodiments only represent a part of the application examples of the present invention and do not limit the technical scope of the present invention to the specific structure of the above embodiments.

[0076] For example, in the first embodiment described above, when the shooting scene of the image data is at night, when the predicted pixel value of the coded object block of the infrared image frame is calculated using the visible light image frame as the reference viewpoint, the luminance component value of the visible light image frame becomes smaller (the image becomes darker), while the luminance component value of the infrared image frame is maintained. Therefore, the luminance component value of the corresponding block of the visible light image frame can be multiplied by a weighting coefficient so that the predicted pixel value of the coded object block of the infrared image frame is greater than the luminance component value of the corresponding block of the visible light image frame.

[0077] Furthermore, for example, in the above embodiments, the computer program executed in the encoding device 100 may be provided in the form of a computer-readable portable recording medium such as a semiconductor memory, magnetic recording medium or optical recording medium, or it may be provided as a computer program product.

[0078] Symbol Explanation

[0079] 20 - Developing unit, 30 - Color space conversion unit, 40 - Encoding processing unit, 100 - Encoding device.

Claims

1. An encoding device for encoding image data of a camera sensor having a three-primary-color filter and an infrared filter in the visible light wavelength region, the encoding device being characterized by comprising: The developing unit develops the image data to generate visible light image frames and infrared image frames. The color space conversion unit converts the color space of the visible light image frame and the infrared image frame from the RGB color space to the YUV color space; and The encoding processing unit encodes the color-converted visible light image frame and the infrared image frame. The encoding processing unit performs the following processing: When predicting the pixel values ​​of the coded object blocks of the visible light image frame and the infrared image frame, a prediction mode is selected from intra-frame prediction, inter-frame prediction, and inter-viewpoint prediction; and The difference between the predicted pixel value obtained by predicting the pixel value of the coded object block of the visible light image frame and the infrared image frame by the prediction mode and the actual pixel value of the coded object block of the visible light image frame and the infrared image frame after color conversion is encoded.

2. The encoding device according to claim 1, characterized in that, The encoding processing unit performs the following processing: When inter-viewpoint prediction is selected as the prediction mode, the disparity vector value is set to zero. A corresponding block that is related to the coded object block in another image frame not selected as the reference viewpoint is searched from the block in one image frame selected as the reference viewpoint in the visible light image frame and the infrared image frame. The predicted pixel value of the coded object block in the other image frame is calculated based on the pixel value of the corresponding block in the image frame selected as the reference viewpoint.

3. The encoding device according to claim 1, characterized in that, The encoding processing unit performs the following processing: When inter-viewpoint prediction is selected as the prediction mode, a corresponding block that is related to the coded object block in another image frame not selected as the reference viewpoint is searched from the blocks in one image frame selected as the reference viewpoint in the visible light image frame and the infrared image frame. The luminance component value of the corresponding block in the image frame selected as the reference viewpoint is set as the predicted pixel value of the luminance component of the coded object block in the other image frame, and the predicted pixel value of the pigment component of the coded object block in the other image frame is set to zero.

4. The encoding device according to claim 2 or 3, characterized in that, The encoding processing unit performs the following processing: Based on the photographic scene of the image data, one of the visible light image frame and the infrared image frame is selected as the reference viewpoint; and When the shooting scene is during the day, the visible light image frame is selected as the reference viewpoint; when the shooting scene is at night, the infrared image frame is selected as the reference viewpoint.

5. A computer program, characterized in that, The computer will perform the following processing: The image data of the camera sensor with a 3-primary-color filter and an infrared filter in the visible light wavelength region are developed to generate visible light image frames and infrared image frames. Convert the color space of the visible light image frame and the infrared image frame from the RGB color space to the YUV color space; When predicting the pixel values ​​of the coded object blocks of the visible light image frame and the infrared image frame, a prediction mode is selected from intra-frame prediction, inter-frame prediction, and inter-viewpoint prediction; and The difference between the predicted pixel value obtained by predicting the pixel value of the coded object block of the visible light image frame and the infrared image frame by the prediction mode and the actual pixel value of the coded object block of the visible light image frame and the infrared image frame after color conversion is encoded.

Images