Information processing device and method

Abstract

The present disclosure relates to an information processing device and a method which make it possible to control a code amount while guaranteeing pixel values of a portion of sensing data. This invention: discriminates, for each pixel value of sensing data, whether the pixel value is a reserved word, which is a predetermined prescribed value; generates a flag map indicating the position of a reserved word pixel, which is a pixel using the reserved word as a pixel value, and non-reserved word data configured from pixel values of non-reserved word pixels, which are pixels that are not the reserved word pixel; reversibly encodes the generated flag map; generates a flag stream; encodes the generated non-reserved word data; and generates a pixel stream. The present disclosure can be applied to, for example, an information processing device, an image processing device, an electronic device, an information processing method, an image processing method, or a program.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that are capable of controlling the code amount while guaranteeing the pixel values ​​of some of the sensing data. [Background technology]

[0002] In recent years, the use of image sensors for sensing applications has been expanding. For example, a technology has been developed to measure the distance to a target object by detecting the phase difference between a sine wave emitted from a light source and a reflected wave (see, for example, Non-Patent Document 1). When using image sensors for such sensing applications, it has been common to combine the output of the image sensor with subsequent signal processing to process it into the desired data and output it, rather than using it directly. As a form of this subsequent signal processing, for example, it has been considered to encode (compress) the sensed data before outputting it. Another method has also been developed that selects an arbitrary color in a map image, separates it into a complementary color file and a selected color flag file, and compresses them separately (see, for example, Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Keita Yasutomi, Shoji Kawato, "Technical Explanation of Time-of-Flight Camera", Journal of the Institute of Image Information and Television Engineers, Vol. 70, No. 11, 2016, pp. 880-885, received August 29, 2016 [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-313569 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when the sensed data contains reserved words with predetermined values, lossy encoding methods may cause the values ​​to change after decoding, making it difficult to correctly restore the reserved words. In contrast, lossless encoding methods may make it difficult to control the code amount. Therefore, it is difficult to control the code amount while ensuring the pixel values ​​of some of the sensed data. The method described in Patent Document 1 also does not take such reserved words into consideration, making it difficult to control the code amount while ensuring the pixel values ​​of some of the sensed data.

[0006] The present disclosure has been made in view of such circumstances, and makes it possible to control the code amount while guaranteeing some pixel values ​​of sensing data. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present technology includes a data discriminator that discriminates whether each pixel value of sensing data is a reserved word, which is a predetermined value, and generates a flag map indicating the positions of reserved word pixels, which are pixels having the reserved word as a pixel value, and non-reserved word data configured of pixel values ​​of non-reserved word pixels, which are pixels that are not the reserved word pixels; a flag encoder that losslessly encodes the flag map generated by the data discriminator and generates a flag stream; and encodes the non-reserved word data generated by the data discriminator. The bit length is the difference between the fixed bit length of the sensing stream and the bit length of the flag stream. a pixel encoding unit for generating a pixel stream; a multiplexing unit that multiplexes the flag stream generated by the flag encoding unit and the pixel stream generated by the pixel encoding unit to generate the sensing stream; The information processing device is provided with:

[0008] An information processing method according to one aspect of the present technology includes: discriminating, for each pixel value of sensing data, whether the pixel value is a reserved word, which is a predetermined value; generating a flag map indicating the positions of reserved word pixels, which are pixels having the reserved word as a pixel value, and non-reserved word data configured by pixel values ​​of non-reserved word pixels, which are pixels that are not the reserved word pixels; losslessly encoding the generated flag map; generating a flag stream; and encoding the generated non-reserved word data; The bit length is the difference between the fixed bit length of the sensing stream and the bit length of the flag stream. Generate pixel stream and multiplexing the generated flag stream and the generated pixel stream to generate the sensing stream. It is an information processing method.

[0009] According to another aspect of the present technology, there is provided an information processing device including: a flag decoding unit that losslessly decodes a flag stream included in a sensing stream and generates a flag map indicating positions of reserved word pixels, which are pixels having pixel values ​​that are predetermined reserved words; a demultiplexing unit that demultiplexes the sensing stream having a fixed bit length and extracts a pixel stream having a bit length that is a difference between the bit length of the fixed length of the sensing stream and the bit length of the flag stream; The information processing device includes a pixel decoding unit that decodes a pixel stream, which is encoded data of non-reserved word data composed of pixel values ​​of non-reserved word pixels, which are pixels that are not reserved word pixels, and generates a pixel map including the non-reserved word data, and a data synthesis unit that synthesizes the flag map generated by the flag decoding unit and the pixel map generated by the pixel decoding unit to generate sensing data.

[0010] An information processing method according to another aspect of the present technology includes losslessly decoding a flag stream included in a sensing stream, and generating a flag map indicating positions of reserved word pixels, which are pixels having pixel values ​​that are predetermined reserved words; demultiplexing the sensing stream having a fixed bit length, and extracting a pixel stream having a bit length equal to the difference between the fixed bit length of the sensing stream and the bit length of the flag stream; the sensing stream Extracted from , is encoded data of non-reserved word data composed of pixel values ​​of non-reserved word pixels, which are pixels that are not reserved word pixels. The aforementioned The information processing method includes decoding a pixel stream, generating a pixel map including the non-reserved word data, and combining the generated flag map with the generated pixel map to generate sensing data.

[0011] In an information processing device and method according to one aspect of the present technology, each pixel value of sensing data is discriminated as to whether it is a reserved word, which is a predetermined value, and a flag map indicating the positions of reserved word pixels, which are pixels having the reserved word as a pixel value, and non-reserved word data consisting of pixel values ​​of non-reserved word pixels, which are pixels that are not reserved word pixels, are generated; the generated flag map is losslessly encoded to generate a flag stream; and the generated non-reserved word data is encoded; The bit length is the difference between the fixed bit length of the sensing stream and the bit length of the flag stream. A pixel stream is generated, The generated flag stream and the generated pixel stream are multiplexed to generate a sensing stream.

[0012] In an information processing device and method according to another aspect of the present technology, a flag stream included in a sensing stream is losslessly decoded, and a flag map indicating positions of reserved word pixels, which are pixels having pixel values ​​that are predetermined reserved words, is generated; The sensing stream having a fixed bit length is demultiplexed, and a pixel stream having a bit length equal to the difference between the fixed bit length of the sensing stream and the bit length of the flag stream is extracted. The sensing stream Extracted from A pixel stream, which is encoded data of non-reserved word data composed of pixel values ​​of non-reserved word pixels, which are pixels that are not reserved word pixels, is decoded, a pixel map including the non-reserved word data is generated, and the generated flag map and the generated pixel map are combined to generate sensing data. [Brief explanation of the drawings]

[0013] [Figure 1] 1A and 1B are diagrams illustrating an example of distance measurement using an indirect ToF method. [Figure 2] 1A and 1B are diagrams illustrating an example of distance measurement using an indirect ToF method. [Figure 3] 1A and 1B are diagrams illustrating an example of distance measurement using an indirect ToF method. [Figure 4] FIG. 10 is a diagram illustrating an example of sensing data. [Figure 5] FIG. 10 is a diagram illustrating an example of sensing data. [Figure 6] FIG. 1 is a block diagram illustrating an example of the main configuration of an encoding device. [Figure 7]FIG. 2 is a block diagram showing an example of the main configuration of a sensing data encoding unit. [Figure 8] FIG. 10 is a diagram illustrating a flag stream. [Figure 9] FIG. 10 is a diagram illustrating an example of a sensing stream. [Figure 10] FIG. 2 is a block diagram illustrating an example of the main configuration of a pixel encoding unit. [Figure 11] FIG. 10 is a diagram illustrating an example of a sensing stream. [Figure 12] 10 is a flowchart illustrating an example of the flow of an encoding process. [Figure 13] 10 is a flowchart showing an example of the flow of a sensing data encoding process. [Figure 14] FIG. 2 is a block diagram illustrating an example of the main configuration of a decoding device. [Figure 15] FIG. 2 is a block diagram showing an example of the main configuration of a sensing data decoding unit. [Figure 16] FIG. 10 is a diagram illustrating an example of the syntax of a flag stream. [Figure 17] FIG. 10 is a diagram illustrating an example of pixel stream syntax. [Figure 18] 10 is a flowchart illustrating an example of the flow of a decoding process. [Figure 19] 10 is a flowchart showing an example of the flow of a sensing data decoding process. [Figure 20] FIG. 10 is a diagram illustrating an example of flag map values. [Figure 21] FIG. 10 is a diagram illustrating an example of pixel stream syntax. [Figure 22] FIG. 2 is a block diagram showing an example of the main configuration of a detection device. [Figure 23] FIG. 1 is a block diagram illustrating an example of the main configuration of a polarization camera. [Figure 24] FIG. 1 is a diagram illustrating a polarization sensor. [Figure 25] FIG. 1 is a block diagram illustrating an example of the main configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. Encoding and decoding of sensing data including reserved words 2. First embodiment (encoding device) 3. Second embodiment (decoding device) 4. Application Examples 5. Additional Notes

[0015] <1. Encoding and decoding of sensing data including reserved words> <itof> In recent years, the use of image sensors for sensing purposes has been increasing. For example, as described in Non-Patent Document 1, a technology has been devised to measure the distance to a target object by detecting the phase difference between a sine wave emitted from a light source and a reflected wave.

[0016] For example, as shown in Fig. 1, ToF (Time-of-Flight) has been considered, in which light (e.g., infrared light) is emitted from a light source 1 toward a subject 3, the time it takes for the reflected light to be received by a distance measuring sensor 2 is measured, and the distance to the subject 3 is derived based on that time. In addition, two ToF methods have been considered: a direct ToF method (also called dToF (direct Time of Flight)) and an indirect ToF method (also called iToF (indirect Time of Flight)).

[0017] In the case of the direct ToF method, it is difficult to increase the number of pixels because a TDC (Time-to-Digital Converter) is used. In the case of the indirect ToF method, a time calculation circuit such as a TDC is not required within the pixel, and an increase in the number of elements within the pixel can be suppressed. Therefore, it is easy to achieve an increase in the number of pixels.

[0018] In the indirect ToF method, the photocharge generated by the photodiode is modulated by a lock-in pixel using a time window (clock) synchronized with the light source. At this time, the time information is reflected in the signal amount, so the time of flight can be derived.

[0019] Modulation methods used in indirect ToF methods include, for example, continuous wave modulation and pulse wave modulation. An example of continuous wave modulation is shown in Figure 2. In the graph shown in Figure 2, sine wave 11 indicates emitted light, and sine wave 12 indicates reflected light. As shown in Figure 2, in the case of continuous wave modulation, the phase difference between emitted light and reflected light can be derived by performing lock-in detection using four time windows. Lock-in detection used here refers to the operation of accumulating signals of the same phase multiple times using a short electronic shutter synchronized with the light source. When modulated with a sine wave, the phase difference φ TOF is derived as shown in the following equation (1) using signals A0, A1, A2, and A3 accumulated in four time windows TW1, TW2, TW3, and TW4.

[0020] TIFF0007757321000001.tif2357

[0021] Modulation frequency f m Since is known, the phase (φ TOF ) to time (t TOF ) can be converted into

[0022] TIFF0007757321000002.tif2328

[0023] The received light contains a DC component other than that of the light source, i.e., a background light component, but this background light component is canceled out by the calculation of the above equation (1). Therefore, as long as the sensor is not saturated, distance can be estimated without being affected by background light.

[0024] In contrast, an example of pulse wave modulation is shown in Figure 3. In the case of pulse wave modulation, if the signals accumulated by TW1 and TW2 are A0 and A1, respectively, the time of flight t TOF is derived as shown in the following equation (3).

[0025] TIFF0007757321000003.tif2439

[0026] TD is an emission window that emits excess background light. By setting the number of time windows TW to three or more, it is possible to know only the background light component, making it possible to estimate distance without being affected by background light. With pulse wave modulation, by setting a high duty ratio for the light source, it is possible to achieve imaging that is robust against background light.

[0027] <Sensing data> In actual sensing, there may be pixels where the light is saturated or where the light cannot be received, resulting in pixels from which the depth cannot be calculated. For example, as shown in FIG. 4A, assume that light (also referred to as irradiated light) is irradiated onto object 20-1 and object 20-2, and an image sensor (not shown) detects the light (also referred to as reflected light) reflected by object 20-1 and object 20-2. Assume that object 20-2 has a much higher reflectivity than object 20-1. When there is no need to distinguish between object 20-1 and object 20-2, they are referred to as object 20. In such a case, if the amount of reflected light is within the tolerance range of the image sensor, it is expressed as a correct pixel value. However, if the amount of reflected light is too strong, the pixel value may be saturated in the image sensor. For example, assume that the amount of reflected light 22, which is the irradiated light 21 reflected by object 20-1 with low reflectivity, is significantly attenuated compared to the irradiated light 21 and falls within the tolerance range of the image sensor. In such a case, the pixel values ​​of the pixels in the image sensor that receive reflected light 22 will not saturate and can correctly represent the amount of reflected light 22. On the other hand, suppose that the amount of reflected light 24, which is the illumination light 23 reflected by object 20-2 with high reflectivity, is not attenuated as much as the illumination light 23 and exceeds the tolerance range of the image sensor. In such a case, the pixel values ​​of the pixels in the image sensor that receive reflected light 24 will saturate and cannot correctly represent the amount of reflected light 24. Furthermore, suppose that illumination light 25 deviates from object 20 and is not reflected. In such a case, the image sensor will not be able to detect the reflected light.

[0028] As a result of this phenomenon, image data 30 detected by the image sensor (i.e., sensed data) becomes, for example, as shown in B in FIG. 4. That is, the pixel values ​​of region 31 (light gray portion) corresponding to object 20-1 in image data 30 correctly represent the intensity of reflected light. In contrast, the pixel values ​​of region 32 (white portion) corresponding to object 20-2 are saturated (so-called blown-out highlights) and do not correctly represent the intensity of reflected light. Furthermore, the pixel values ​​of the remaining region 33 (dark gray portion) are approximately zero. That is, in this image data, the correct distance to object 20 can only be determined from region 31.

[0029] Therefore, it was conceived to represent the light receiving state of a pixel by assigning a predetermined reserved word to a pixel whose pixel value is saturated or which cannot detect light, as in the sensing data 40 shown in FIG. 5. In the example of FIG. 5, the detected pixel value is set as is to the pixel in the light gray area on the left side of the sensing data 40. In contrast, the reserved word "998" is set to the pixel in the white area on the upper right. Furthermore, the reserved word "999" is set to the pixel in the dark gray area on the lower right. The reserved word "998" indicates that the pixel value is saturated (e.g., reaches its maximum value), while the reserved word "999" indicates that light cannot be received (e.g., the pixel value is approximately zero). In other words, some meaningless pixel value is set to a pixel whose pixel value is saturated or which cannot receive light when the image sensor detects light, but these pixel values ​​are replaced with predetermined reserved words.

[0030] By doing so, pixels with reserved words can be easily ignored when calculating distance. In other words, pixels with saturated pixel values ​​or pixels that could not receive light, which make it impossible to calculate an accurate distance, can be easily ignored when calculating distance. This prevents unnecessary processing from increasing. Furthermore, this type of substitution can reduce the amount of code generated when encoding sensing data. For example, taking into account errors and the like, a value lower than the hardware limit of the image sensor can be set as a "saturated pixel value." In such cases, the "saturated pixel value" can take multiple values. Furthermore, the pixel value that is determined to have "not received light" does not have to be "zero" but can be, for example, an extremely small pixel value (a value between zero and a predetermined range). In such cases, the pixel value that is determined to have "not received light" can take multiple values. In other words, "meaningless pixel values" can vary among pixels. When sensing data containing such meaningless pixel values ​​is encoded, this variation can result in an increased amount of code generated. By replacing these meaningless pixel values ​​with reserved words as described above, the variation in pixel values ​​can be reduced. For example, in the case of the example of the sensing data 40 in Fig. 5, meaningless pixel values ​​are replaced with two types of reserved words (two types of pixel values). In other words, when meaningless pixel values ​​are composed of three or more types of pixel values, such replacement can suppress the variation in pixel values. Therefore, the amount of code generated when encoding the sensing data can be reduced.

[0031] The layout of each area in the example of FIG. 5 is an example and does not correspond to the example of FIG.

[0032] <Sensing data encoding> When using image sensors for such sensing applications, it is common to combine the output of an image sensor with subsequent signal processing to process it into the desired data and output it, rather than using it directly. One possible method of subsequent signal processing is to encode (compress) the sensed data before outputting it. For example, reducing the amount of information can be considered to reduce the bandwidth required when inputting and outputting the sensed data via an interface. When storing the sensed data in a storage medium, reducing the amount of information can be considered to reduce the storage capacity. Furthermore, when transmitting the sensed data via a communication path (transmission medium), reducing the amount of information can be considered to reduce the bandwidth used in the communication path. In such cases, it is necessary to be able to control the amount of information to a desired value.

[0033] For example, when encoding sensor data using a lossless method, the compressed data length depends on the input data. In other words, the amount of information may increase when the data is encoded. For this reason, it has been difficult to apply lossless encoding methods to control the amount of information as described above.

[0034] In contrast, when encoding sensed data using a lossy method, the compression rate can be adjusted as desired, making it easy to control the amount of information as desired. However, encoding can cause degradation and pixel values ​​to change. For pixel values ​​that can be used for distance measurement, only the measurement result (distance) changes depending on the pixel value, so slight degradation is acceptable.

[0035] However, even a slight change in the value of a reserved word can change its meaning. For example, in the sensing data 40 of FIG. 5, if the reserved word "998" changes to "999," the sensing data 40 changes from "saturated pixel" to "pixel where reflected light cannot be detected." There is also a risk that a reserved word may no longer be a reserved word. Furthermore, there is also a risk that a non-reserved word may become a reserved word. In this way, if the meaning of the pixel value of the sensing data changes, there is a risk of problems occurring in image processing, etc., that uses this sensing data.

[0036] Therefore, it has been difficult to apply a lossy encoding method to the above-mentioned control of the amount of information. In other words, whether encoding the sensed data using a lossless method or a lossy method, it has been difficult to control the amount of code while preserving some of the pixel values ​​of the sensed data.

[0037] For example, Patent Document 1 proposes a method of selecting any color in a map image, separating it into a complementary color file and a selected color flag file, and compressing them separately. However, separating files like this method risks increasing the amount of processing and data, making it difficult to apply to the information volume control described above. Furthermore, Patent Document 1 does not take reserved words into consideration, and does not disclose a method for identifying reserved words contained in sensing data and processing them separately from other pixel values. In other words, the method described in Patent Document 1 makes it difficult to control the amount of code while preserving some pixel values ​​of the sensing data.

[0038] <Encoding control for reserved and non-reserved words> Therefore, each pixel value of the sensing data is discriminated as to whether it is a reserved word or not, and information relating to the reserved word is encoded in a lossless manner.

[0039] For example, in an information processing method, each pixel value of sensing data is discriminated as to whether it is a reserved word, which is a predetermined value, and a flag map indicating the positions of reserved word pixels, which are pixels whose pixel values ​​are reserved words, and non-reserved word data consisting of pixel values ​​of non-reserved word pixels, which are pixels that are not reserved word pixels, are generated, the flag map is losslessly encoded to generate a flag stream, and the non-reserved word data is encoded to generate a pixel stream.

[0040] For example, an information processing device may include a data discriminator that discriminates whether each pixel value of sensing data is a reserved word, which is a predetermined value, and generates a flag map indicating the positions of reserved word pixels, which are pixels whose pixel value is the reserved word, and non-reserved word data consisting of pixel values ​​of non-reserved word pixels, which are pixels that are not reserved word pixels; a flag encoder that losslessly encodes the flag map generated by the data discriminator to generate a flag stream; and a pixel encoder that encodes the non-reserved word data generated by the data discriminator to generate a pixel stream.

[0041] For example, in an information processing method, a flag stream included in a sensing stream is losslessly decoded to generate a flag map indicating the positions of reserved word pixels, which are pixels whose pixel values ​​are predetermined reserved words; a pixel stream, which is encoded data of non-reserved word data composed of pixel values ​​of non-reserved word pixels, which are pixels that are not reserved word pixels, included in the sensing stream, is decoded; a pixel map including the non-reserved word data is generated; and the generated flag map and the generated pixel map are combined to generate sensing data.

[0042] For example, an information processing device may include a flag decoding unit that losslessly decodes a flag stream included in a sensing stream and generates a flag map indicating the positions of reserved word pixels, which are pixels whose pixel values ​​are predetermined reserved words; a pixel decoding unit that decodes a pixel stream that is encoded data of non-reserved word data composed of pixel values ​​of non-reserved word pixels, which are pixels that are not reserved word pixels, included in the sensing stream, and generates a pixel map including the non-reserved word data; and a data synthesis unit that synthesizes the flag map generated by the flag decoding unit and the pixel map generated by the pixel decoding unit to generate sensing data.

[0043] In this way, the sensing data can be compressed to a desired amount of information without degrading the reserved words. In other words, the amount of code can be controlled while guaranteeing the pixel values ​​of some of the sensing data.

[0044] This allows, for example, the amount of information to be reduced in order to suppress the bandwidth required when inputting and outputting sensing data via an interface. Furthermore, when storing sensing data in a storage medium, the amount of information can be reduced in order to suppress the storage capacity. Furthermore, when transmitting sensing data via a communication path (transmission medium), the amount of information can be reduced in order to suppress the bandwidth used in the communication path.

[0045] 2. First Embodiment <Encoding device> Fig. 6 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device to which the present technology is applied. The encoding device 100 shown in Fig. 6 is a device that encodes sensing data. Note that Fig. 6 shows main processing units, data flows, etc., and does not necessarily show everything. In other words, in this encoding device 100, there may be processing units that are not shown as blocks in Fig. 6, or there may be processing or data flows that are not shown as arrows, etc. in Fig. 6.

[0046] As shown in FIG. 6, the encoding device 100 includes a block dividing unit 101, a sensing data encoding unit 102, and an output unit 103.

[0047] The block division unit 101 acquires sensing data generated by another device. The sensing data is data indicating the detection results of some sensor. For example, it may be image data detected by an image sensor. In the following, the sensing data will be described as indirect ToF (iToF) sensing data as described with reference to FIGS. 1 to 5. In this case, the ToF sensor that receives reflected light has multiple pixels, similar to the case of an image sensor, and is capable of detecting reflected light for each pixel. For example, an image sensor may be used as the ToF sensor. In other words, the sensing data is composed of the detection results (pixel values) of each of the multiple pixels, like image data composed of multiple pixel values.

[0048] The block dividing unit 101 divides the acquired sensing data into units of encoding processing (also referred to as blocks) in the sensing data encoding unit 102. The block dividing unit 101 supplies the sensing data to the sensing data encoding unit 102 for each block.

[0049] The sensing data encoding unit 102 acquires the sensing data supplied for each block from the block division unit 101. The sensing data encoding unit 102 encodes the acquired sensing data for each block to generate a sensing stream. The sensing data encoding unit 102 supplies the generated sensing stream to the output unit 103.

[0050] The output unit 103 is an output interface that outputs the sensing stream. The output unit 103 acquires the sensing stream supplied from the sensing data encoding unit 102 and outputs it to the outside of the encoding device 100. The sensing stream is, for example, transmitted to another information processing device or the like via a communication path, or supplied to a storage medium and stored therein.

[0051] In the encoding device 100 configured as described above, the present technology is applied to the sensing data encoding unit 102. That is, as described above in <Respective encoding control of reserved words and non-reserved words>, the sensing data encoding unit 102 discriminates whether each pixel value of the sensing data is a reserved word or not, and encodes information related to the reserved words in a lossless manner. In this way, it is possible to compress the sensing data to a desired amount of information without degrading the reserved words. That is, it is possible to control the amount of code while guaranteeing the pixel values ​​of some of the sensing data.

[0052] <Sensing data encoding unit> Fig. 7 is a block diagram showing an example of the main configuration of the sensing data encoding unit 102 in Fig. 6. Note that Fig. 7 shows the main processing units, data flows, etc., and is not limited to what is shown in Fig. 7. In other words, in this sensing data encoding unit 102, there may be processing units that are not shown as blocks in Fig. 7, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 7.

[0053] As shown in FIG. 7, the sensing data encoding unit 102 includes a data discriminator 131, a flag encoder 132, a pixel encoder 133, and a multiplexer 134.

[0054] The data discriminator 131 acquires sensing data for each block supplied from the block divider 101. The data discriminator 131 separates the acquired sensing data into data to be losslessly encoded (data to be compressed losslessly) and data to be lossy encoded (data to be compressed lossily). In other words, the data discriminator 131 refers to each pixel value of the sensing data and discriminates whether or not the value is a reserved word, which is a predetermined value.

[0055] Then, based on the discrimination results, the data discriminator 131 generates a flag map that indicates the positions of reserved word pixels, which are pixels that have reserved words as pixel values. The flag map is array data configured with indexes (identification information) that indicate whether each pixel is a reserved word pixel or not (and, if there are multiple reserved words, which reserved word the reserved word pixel corresponds to). The number of array elements is the same as the number of pixels to be processed, NumPixels, and each array element indicates whether the pixel at the same position is a normal pixel or an invalid flag. N can be expressed as in the following equation (4):

[0056] TIFF0007757321000004.tif1159...(4)

[0057] Furthermore, the data discriminator 131 generates image data, which is data obtained by excluding pixel values ​​of reserved word pixels from the sensing data, based on the discrimination results. In other words, this image data is composed of pixel values ​​of non-reserved word pixels, which are pixels that are not reserved word pixels. Therefore, this image data is also referred to as non-reserved word data.

[0058] The data discriminator 131 supplies the generated flag map to the flag encoder 132 and the pixel encoder 133. The data discriminator 131 supplies the generated image data (non-reserved word data) to the pixel encoder 133.

[0059] The flag encoding unit 132 acquires the flag map supplied from the data discriminator 131. The flag encoding unit 132 encodes the acquired flag map in a lossless manner to generate a flag stream. This encoding method may be any as long as it is lossless.

[0060] For example, the flag encoding unit 132 may include in this flag stream flag information IsFlagExist indicating whether the reserved word pixel is included in the flag map. The flag information IsFlagExist is flag information generated for each block and indicates whether the corresponding block (the block to be processed) includes a reserved word pixel. An example of the semantics of the flag information IsFlagExist is shown in A of FIG. 8. If the value of the flag information IsFlagExist is true (e.g., "1"), this indicates that the block to be processed includes a reserved word pixel. If the value of the flag information IsFlagExist is false (e.g., "0"), this indicates that the block to be processed does not include a reserved word pixel, i.e., it includes only non-reserved word pixels.

[0061] If the flag information IsFlagExist is false, the flag encoding unit 132 generates a flag stream consisting only of the flag information IsFlagExist. For example, as shown in B of FIG. 8, it is assumed that the flag map 141 is composed only of indices (circled "1" in the figure) indicating normal pixels (non-reserved word pixels). In this case, the flag encoding unit 132 generates a flag stream 142 consisting only of the flag information IsFlagExist. The code length of the flag stream 142 is 1 bit (the value is "0").

[0062] On the other hand, if the flag information IsFlagExist is true, the flag encoding unit 132 generates a flag stream consisting of the flag information IsFlagExist and a flag map. For example, as shown in C of FIG. 8, it is assumed that the flag map 143 contains an index indicating a reserved word pixel (circled "2" or "3" in the figure). In this case, the flag encoding unit 132 generates a flag stream 144 consisting of the flag information IsFlagExist and the flag map. Since the flag map contains information for 8 pixels, if the index is 2 bits, the code length of the flag stream 144 is 17 bits (IsFlagExist 1 bit + 2 bits × 8 pixels).

[0063] In the example of FIG. 8, the flag map index is made up of two bits (00, 01, 10) to distinguish between three states: normal pixel, saturated, and low light sensitivity. The bit length of the flag map (index) is arbitrary. As in the example of FIG. 8, the flag map (index) may have a fixed length.

[0064] 7, the flag encoding unit 132 supplies the generated flag stream to the multiplexing unit 134. In addition, the flag encoding unit 132 supplies information indicating the bit length (code amount) of the flag stream (Flag Stream Bit Length) to the pixel encoding unit 133.

[0065] The pixel encoding unit 133 acquires the image data and flag map supplied from the data discriminator 131. The pixel encoding unit 133 also acquires information indicating the bit length (code amount) of the flag stream supplied from the flag encoding unit 132.

[0066] The pixel encoding unit 133 encodes the image data based on the flag map and information indicating the bit length (code amount) of the flag stream to generate a pixel stream. That is, the pixel encoding unit 133 encodes only the pixel values ​​of non-reserved word pixels among the pixel values ​​of the sensing data.

[0067] The bit length (code length) of this sensing stream may be a fixed length.

[0068] In this case, the pixel encoding unit 133 may generate a pixel stream whose bit length (code length) is the difference between the fixed bit length (code length) of the sensing stream and the bit length (code length) of the flag stream.

[0069] For example, as shown in FIG. 9, the code length of a sensing stream consisting of a flag stream and a pixel stream is fixed to a code length specified by the user. In this way, the code length of the sensing stream can be fixed. This allows the desired sensing stream to have the desired code length. At this time, the pixel stream is generated so that it has the remaining code length excluding the code length of the losslessly encoded flag stream. For example, if many bits are required for flag data, the code length of the image data (pixel stream) can be reduced, and if the flag data can be compressed with a small amount of data, the code length of the image data (pixel stream) can be increased accordingly. In this way, the losslessness of the flag stream can be guaranteed.

[0070] In this case, the pixel encoding unit 133 may set the code length of the pixel stream based on information indicating the bit length (code amount) of the flag map and the flag stream. In this way, the pixel encoding unit 133 can more easily control the code amount of the pixel stream.

[0071] Alternatively, the flag encoding unit 132 may supply flag information IsFlagExist to the pixel encoding unit 133 as information indicating the bit length (code amount) of the flag stream (Flag Stream Bit Length), and the pixel encoding unit 133 may set the bit length (code amount) of the pixel stream based on the flag information IsFlagExist. For example, if the flag map index has a fixed length, the code amount of the flag stream may be determined by the value of the flag information IsFlagExist.

[0072] Furthermore, since reserved words are removed from the image data, its size is variable (it can be smaller than the size of the sensing data). In other words, the size of the blocks processed by the pixel encoding unit 133 is variable. This block size may be fixed by inserting dummy pixel values ​​into the positions of reserved word pixels. In this case, the pixel encoding unit 133 simply identifies the positions of the reserved word pixels based on the flag map and inserts dummy pixel values ​​into the positions of the reserved word pixels in the image data.

[0073] Fig. 10 is a block diagram showing an example of the main configuration of the pixel encoding unit 133. Note that Fig. 10 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 10. In other words, in this pixel encoding unit 133, there may be processing units that are not shown as blocks in Fig. 10, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 10.

[0074] In the example of FIG. 10, the pixel encoding unit 133 includes a reserved word pixel processing unit 161, a 6-bit length encoding unit 162, an 8-bit length encoding unit 163, and a selection unit 164.

[0075] The reserved word pixel processing unit 161 acquires image data supplied from the data discriminator 131. The reserved word pixel processing unit 161 acquires a flag map supplied from the data discriminator 131. The reserved word pixel processing unit 161 performs processing on reserved word pixels based on the flag map. For example, the reserved word pixel processing unit 161 inserts dummy pixel values ​​into the reserved word pixel positions of the image data based on the flag map. The reserved word pixel processing unit 161 supplies the processed image data to the 6-bit length coding unit 162.

[0076] The 6-bit length coding unit 162 acquires image data supplied from the reserved word pixel processing unit 161. The 6-bit length coding unit 162 encodes each pixel value of the acquired image data with a 6-bit length. In other words, the 6-bit length coding unit 162 generates a pixel stream with a code amount of 6 bits x number of pixels. The 6-bit length coding unit 162 supplies the generated pixel stream to the selection unit 164.

[0077] The 8-bit length coding unit 163 acquires image data supplied from the data discriminator 131. The 8-bit length coding unit 163 encodes each pixel value of the acquired image data with an 8-bit length. In other words, the 8-bit length coding unit 163 generates a pixel stream with a code amount of 8 bits x number of pixels. The 8-bit length coding unit 162 supplies the generated pixel stream to the selector 164.

[0078] The selection unit 164 acquires flag information IsFlagExist supplied from the flag encoding unit 132. The selection unit 164 acquires the pixel stream supplied from the 6-bit length encoding unit 162. The selection unit 164 acquires the pixel stream supplied from the 8-bit length encoding unit 163. Based on the value of the flag information IsFlagExist, the selection unit 164 outputs (supplies to the multiplexing unit 134) either the pixel stream supplied from the 6-bit length encoding unit 162 or the pixel stream supplied from the 8-bit length encoding unit 163. In other words, if the flag information IsFlagExist is true, the selection unit 164 selects and outputs the pixel stream supplied from the 6-bit length encoding unit 162. On the other hand, if the flag information IsFlagExist is false, the selection unit 164 selects and outputs the pixel stream supplied from the 8-bit length encoding unit 163.

[0079] In other words, if the flag information IsFlagExist is true, the image data is processed in the reserved word pixel processing unit 161 and coded by the 6-bit length coding unit 162 to generate a pixel stream. If the flag information IsFlagExist is false, the image data is coded by the 8-bit length coding unit 163 to generate a pixel stream. In this way, the code amount of the pixel stream can be controlled according to the value of the flag information IsFlagExist.

[0080] The 6-bit length encoding unit 162 and the 8-bit length encoding unit 163 may use any encoding method. Generally, the amount of code can be controlled more reliably by applying a lossy encoding method. However, a lossless encoding method may also be used as long as it can reliably control the amount of code to a desired value.

[0081] Returning to FIG. 7, the pixel encoding unit 133 supplies the generated pixel stream to the multiplexing unit .

[0082] The multiplexing unit 134 acquires the flag stream supplied from the flag encoding unit 132. The multiplexing unit 134 acquires the pixel stream supplied from the pixel encoding unit 133. The multiplexing unit 134 multiplexes the flag stream and the pixel stream to generate a sensing stream. The multiplexing unit 134 supplies the generated sensing stream to the output unit 103 (FIG. 6).

[0083] Fig. 11 is a diagram showing an example of a sensing stream when the pixel encoding unit 133 performs control as in the example of Fig. 10. When the flag information IsFlagExist is true (for example, "1"), the sensing stream is made up of a flag stream including the flag information IsFlagExist and a flag map, and a pixel stream encoded in 6 bits, as shown in A of Fig. 11. On the other hand, when the flag information IsFlagExist is false (for example, "0"), the sensing stream is made up of a flag stream including the flag information IsFlagExist and a pixel stream encoded in 8 bits, as shown in B of Fig. 11.

[0084] When the flag information IsFlagExist is true (for example, "1"), the code length of the flag stream is larger than when the flag information IsFlagExist is false (for example, "0"). Therefore, in this case, as shown in A of FIG. 11, the image data is encoded so that the code length of the pixel stream is smaller. In other words, when the flag information IsFlagExist is false (for example, "0"), the code length of the flag stream is smaller than when the flag information IsFlagExist is true (for example, "1"). Therefore, in this case, as shown in B of FIG. 11, the image data is encoded so that the code length of the pixel stream is larger.

[0085] As described above, the flag encoding unit 132 encodes the flag map in a lossless manner, thereby guaranteeing the integrity of the flag map (its index). In other words, the index value does not change as a result of encoding. As shown in the example of FIG. 11, the code length of the pixel stream can be controlled according to the code length of the flag stream. Therefore, the encoding device 100 can control the code amount while guaranteeing some pixel values ​​of the sensing data.

[0086] <Encoding process flow> An example of the flow of the encoding process executed by the encoding device 100 will be described with reference to the flowchart of FIG.

[0087] When the encoding process starts, in step S101, the block dividing unit 101 of the encoding device 100 divides the sensing data in units of frames into units of blocks (units of encoding).

[0088] In step S102, the sensing data encoding unit 102 executes a sensing data encoding process, encodes the sensing data divided in step S101 for each block, and generates a sensing stream.

[0089] In step S103, the output unit 103 outputs the sensing stream generated in step S102 to the outside of the encoding device 100.

[0090] When the process of step S103 is completed, the encoding process ends.

[0091] <Sensing data encoding process flow> An example of the flow of the sensing data encoding process executed in step S102 of Fig. 12 will be described with reference to the flowchart of Fig. 13. In Fig. 13, a case where a fixed-length sensing stream (a sensing stream with a predetermined code amount) is generated will be described.

[0092] When the sensing data encoding process starts, in step S131, the data discriminator 131 of the sensing data encoder 102 discriminates between normal pixel data and reserved word pixel data. That is, the data discriminator 131 refers to each pixel value of the sensing data and discriminates whether or not the pixel value is a reserved word, which is a predetermined value. Then, based on the discrimination result, the data discriminator 131 generates a flag map indicating the positions of reserved word pixels, which are pixels whose pixel values ​​are reserved words. Furthermore, based on the discrimination result, the data discriminator 131 generates image data, which is data obtained by excluding the pixel values ​​of the reserved word pixels from the sensing data.

[0093] In step S132, the flag encoding unit 132 losslessly encodes the flag map generated in step S131 to generate a flag stream.

[0094] In step S133, the pixel encoding unit 133 sets the bit length (code length) of the pixel stream based on the bit length (code length) of the flag stream generated in step S132. For example, the pixel encoding unit 133 sets the difference between the bit length (code length) of the sensing stream and the bit length (code length) of the flag stream as the bit length (code length) of the pixel stream.

[0095] In step S134, the pixel encoding unit 133 encodes the image data generated in step S131 to generate a pixel stream having the bit length (code length) set in step S133. For example, the pixel encoding unit 133 performs lossy encoding on the image data to generate a pixel stream having the bit length (code length) set in step S133.

[0096] In step S135, the flag stream generated in step S132 and the pixel stream generated in step S134 are multiplexed to generate a fixed-length sensing stream.

[0097] When the process of step S135 ends, the process returns to FIG.

[0098] By performing each process as described above, the encoding device 100 can control the code amount while guaranteeing some pixel values ​​of the sensing data.

[0099] 3. Second Embodiment <Decryption device> Fig. 14 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an information processing device to which the present technology is applied. The decoding device 200 shown in Fig. 14 is a device that decodes a sensing stream. For example, the decoding device 200 can decode a sensing stream generated by encoding sensing data by the encoding device 100 (Fig. 6) to generate sensing data.

[0100] Note that Fig. 14 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in this decoding device 200, there may be processing units that are not shown as blocks in Fig. 14, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 14.

[0101] As shown in FIG. 14, the decoding device 200 includes an input unit 201, a sensing data decoding unit 202, and a block reconstruction unit 203.

[0102] The input unit 201 acquires a sensing stream input to the decoding device 200. This sensing stream is the same information as described in the first embodiment. The input unit 201 supplies the acquired sensing stream to the sensing data decoding unit 202.

[0103] The sensing data decoding unit 202 acquires the sensing stream supplied from the input unit 201. The sensing data decoding unit 202 decodes the sensing stream and generates (reconstructs) sensing data for each block. The sensing data decoding unit 202 supplies the generated sensing data for each block to the block reconstruction unit 203.

[0104] The block reconstruction unit 203 acquires the sensing data for each block supplied from the sensing data decoding unit 202. The block reconstruction unit 203 reconstructs the sensing data for each block to generate sensing data for each frame. The block reconstruction unit 203 outputs the generated sensing data for each frame to the outside of the decoding device 200. This sensing data is supplied to, for example, a processing unit at a subsequent stage and processed.

[0105] In the decoding device 200 configured as described above, the present technology is applied to the sensing data decoding unit 202. That is, as described above in <Respective encoding control of reserved words and non-reserved words>, the sensing data decoding unit 202 decodes encoded data of information related to reserved words in a lossless manner. In this way, it is possible to correctly decode encoded data in which the sensing data has been compressed to a desired amount of information without degrading the reserved words. That is, it is possible to control the amount of code while guaranteeing the pixel values ​​of some of the sensing data.

[0106] <Sensing data encoding unit> Fig. 15 is a block diagram showing an example of the main configuration of the sensing data decoding unit 202 in Fig. 14. Note that Fig. 15 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in this sensing data decoding unit 202, there may be processing units that are not shown as blocks in Fig. 15, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 15.

[0107] As shown in FIG. 15, the sensing data decoding unit 202 includes a flag decoding unit 231 , a demultiplexing unit 232 , a pixel decoding unit 233 , and a data combining unit 234 .

[0108] The flag decoding unit 231 acquires the sensing stream supplied from the input unit 201. The flag decoding unit 231 losslessly decodes the flag stream included in the sensing stream to generate a flag map. This decoding method may be any lossless method that corresponds to the encoding method of the flag stream. For example, the flag decoding unit 231 extracts the flag stream from the sensing stream and losslessly decodes the extracted flag stream.

[0109] In this case, the flag decoding unit 231 may losslessly decode the flag stream to generate flag information IsFlagExist, and generate a flag map according to the value of the flag information IsFlagExist. For example, if the sensing stream is configured as in the example of FIG. 11, the flag decoding unit 231 can obtain the flag information IsFlagExist from the beginning of the flag stream (sensing stream). If the flag information IsFlagExist is false, the flag stream does not include a flag map, so the flag decoding unit 231 generates a flag map in which all pixels are not reserved word pixels. If the flag information IsFlagExist is true, the flag stream includes a flag map, so the flag decoding unit 231 losslessly decodes the flag stream to generate a flag map.

[0110] FIG. 16 shows an example of the syntax of a flag stream. In this example, if the flag information IsFlagExist is true, flag map elements corresponding to the number of pixels to be processed NumPixels are read from the sensing stream. If the flag information IsFlagExist is false, reading of this flag map is omitted. In this case, the flag decoding unit 231 generates a flag map in which all pixels are not reserved word pixels. Note that if the block size is fixed, the number of pixels to be processed NumPixels is known. Alternatively, the block size may be variable and the number of pixels to be processed NumPixels may be signaled (transmitted from the encoding side).

[0111] As described above in the first embodiment, this flag map may be information of a fixed length as in the example of FIG.

[0112] Returning to FIG. 15, the flag decoding unit 231 supplies the generated flag map to the data synthesis unit 234.

[0113] Furthermore, the flag decoding unit 231 may supply the demultiplexing unit 232 with information indicating the bit length (code amount) of the flag stream (Flag Stream Bit Length).

[0114] The demultiplexing unit 232 acquires the sensing stream supplied from the input unit 201. The demultiplexing unit 232 demultiplexes the sensing stream and extracts the pixel stream included in the sensing stream. For example, the demultiplexing unit 232 skips the bit length (code amount) of the flag stream from the beginning of the sensing stream and extracts the subsequent data to extract the pixel stream.

[0115] The bit length (code length) of the sensing stream may be fixed. In this case, the demultiplexing unit 232 may extract, from the sensing stream, a pixel stream whose bit length is the difference between the fixed bit length of the sensing stream and the bit length of the flag stream.

[0116] The demultiplexing unit 232 may also acquire information indicating the bit length (code amount) of the flag stream (Flag Stream Bit Length) supplied from the flag decoding unit 231, demultiplex the sensing stream based on the information, and extract the pixel stream included in the sensing stream. For example, the demultiplexing unit 232 may extract the pixel stream by skipping the bit length indicated in the acquired information from the beginning of the sensing stream and extracting the data thereafter.

[0117] Furthermore, the flag decoding unit 231 may supply flag information IsFlagExist to the demultiplexing unit 232 as information indicating the bit length (code amount) of the flag stream. In this case, the demultiplexing unit 232 may acquire the flag information IsFlagExist and extract a pixel stream from the sensing stream based on the value of the flag information IsFlagExist. For example, the demultiplexing unit 232 may determine the bit length (code amount) of the flag stream based on the value of the flag information IsFlagExist, skip that bit length from the beginning of the sensing stream, and extract the subsequent data to extract the pixel stream.

[0118] The demultiplexer 232 supplies the pixel stream to a pixel decoder 233 .

[0119] The pixel decoding unit 233 acquires the pixel stream supplied from the demultiplexing unit 232. The pixel decoding unit 233 decodes the pixel stream to generate a pixel map. In this case, the pixel decoding unit 233 applies a decoding method corresponding to the encoding method of the pixel stream. For example, if the pixel stream is lossy encoded, the pixel decoding unit 233 lossy decodes the pixel stream. Generally, applying such a lossy decoding method (encoding method) makes it possible to more reliably control the code amount. However, a lossless method may also be applied as long as it can reliably control the code amount to a desired value. In other words, if the pixel stream is losslessly encoded, the pixel decoding unit 233 losslessly decodes the pixel stream.

[0120] FIG. 17 shows an example of the syntax of a pixel stream. For example, if the flag information IsFlagExist is true, decoding according to MIPI CSI-2 Annex E 12-6-12 is applied. MIPI CSI-2 Annex E 12-6-12 is an example of a 6-bit length decoding method. In this case, as in the example described in the first embodiment, it is assumed that the pixel value of each pixel is encoded using a 6-bit length encoding method. The pixel decoding unit 233 can correctly decode the pixel stream by decoding using the 6-bit length decoding method.

[0121] Furthermore, if the flag information IsFlagExist is false, decoding according to MIPI CSI-2 Annex E 12-8-12 is applied. MIPI CSI-2 Annex E 12-8-12 is an example of an 8-bit length decoding method. In this case, as in the example described in the first embodiment, it is assumed that the pixel values ​​of each pixel are encoded using an 8-bit length encoding method. The pixel decoding unit 233 can correctly decode the pixel stream by decoding using the 8-bit length decoding method. Of course, the decoding method may be any method that corresponds to the encoding method of the pixel stream, and is not limited to the above example.

[0122] The pixel map is image data in a state where predetermined processing has been performed on the positions of reserved word pixels. For example, if dummy pixel values ​​are inserted into the positions of reserved word pixels in image data during encoding, this pixel map is image data in a state where the dummy pixel values ​​have been inserted. Note that if image data is encoded as is (without predetermined processing being performed on the positions of reserved word pixels), this pixel map is equivalent to that image data. The pixel decoding unit 233 supplies the pixel map to the data synthesis unit 234.

[0123] The data synthesis unit 234 acquires the flag map supplied from the flag decoding unit 231. The data synthesis unit 234 acquires the pixel map supplied from the pixel decoding unit 233. The data synthesis unit 234 synthesizes the flag map and pixel map to generate sensing data.

[0124] For example, if the pixel map includes the above-mentioned dummy pixel values, the data synthesis unit 234 replaces each dummy pixel value in the pixel map with a reserved word indicated by the index corresponding to that pixel in the flag map. If the pixel map does not include the above-mentioned dummy pixel values, the data synthesis unit 234 inserts a reserved word indicated by the index in the flag map into the position of the reserved word pixel in the pixel map based on the flag map. In this way, the data synthesis unit 234 generates sensing data including a reserved word. If the block does not include a reserved word, the data synthesis unit 234 uses the pixel map as sensing data. The data synthesis unit 234 supplies the sensing data generated as described above (sensing data for each block) to the block reconstruction unit 203.

[0125] As described above, the flag decoding unit 231 decodes the flag stream in a lossless manner, thereby guaranteeing the flag map (index). In other words, the index value does not change due to decoding. As shown in the example of FIG. 11, the code length of the pixel stream can be controlled according to the code length of the flag stream. Therefore, the decoding device 200 can control the code amount while guaranteeing some pixel values ​​of the sensing data.

[0126] <Decryption process flow> An example of the flow of the decoding process executed by the decoding device 200 will be described with reference to the flowchart of FIG.

[0127] When the decoding process starts, the input unit 201 of the decoding device 200 acquires a sensing stream in step S201.

[0128] In step S202, the sensing data decoding unit 202 executes sensing data decoding processing to decode the sensing stream for each block and generate sensing data.

[0129] In step S203, the block reconstructing unit 203 reconstructs the sensing data for each block generated in step S202 to generate sensing data for each frame.

[0130] When the process of step S203 ends, the decoding process ends.

[0131] <Sensing data decoding process flow> An example of the flow of the sensing data decoding process executed in step S202 of FIG. 18 will be described with reference to the flowchart of FIG.

[0132] When the sensing data decoding process starts, in step S231, the flag decoding unit 231 extracts a flag stream from the sensing stream, losslessly decodes the flag stream, and generates a flag map.

[0133] In step S232, the demultiplexer 232 demultiplexes the sensing stream and extracts a pixel stream.

[0134] In step S233, the pixel decoding unit 233 decodes the pixel stream extracted in step S232 to generate a pixel map. For example, the pixel decoding unit 233 lossy-decodes the lossy-encoded pixel stream to generate a pixel map.

[0135] In step S234, the data synthesis unit 234 synthesizes the flag map generated in step S231 and the pixel map generated in step S233 to generate sensing data.

[0136] When the process of step S234 ends, the process returns to FIG.

[0137] By performing each process as described above, the decoding device 200 can control the code amount while guaranteeing the pixel values ​​of some of the sensing data.

[0138] <4. Application Examples> <variable length map flag> The flag map (or its index) may be variable-length information. For example, the flag map may be a variable-length code according to the frequency of occurrence. An example of the semantics of the flag map is shown in FIG. 20. In this example, the index indicating a normal pixel may be "0", the index indicating saturation of the pixel value may be "10", and the index indicating low light-receiving sensitivity may be "110". In this way, by making the code length variable according to the frequency of occurrence, for example, it is possible to improve the coding efficiency.

[0139] <Another example of pixel stream> Alternatively, information about reserved word pixels may not be transmitted in the pixel stream, and the amount of code assigned to each pixel value may be controlled according to the number of reserved word pixels.

[0140] For example, suppose three of the eight pixels are reserved word pixels, as shown on the left side of A in Figure 21. In this case, as shown on the right side, only the information of the five non-reserved word pixels is coded. In other words, in this case, the more reserved word pixels contained in a block, the fewer pixels are coded. Therefore, the amount of code assigned to each pixel is increased accordingly.

[0141] An example of the syntax for this case is shown in B of Figure 21. In this example, 6-bit coding is applied when the reserved word pixel is 1 pixel or 2 pixels, 7-bit coding is applied when the reserved word pixel is 3 pixels, and 8-bit coding is applied when the reserved word pixel is 4 pixels or more. By doing this, it is possible to suppress degradation of pixel values ​​of non-reserved words due to encoding.

[0142] <Application example 1> The present technology described above can be applied to any configuration. For example, the encoding device 100 and the decoding device 200 described above can be applied to any device or system.

[0143] Fig. 22 is a block diagram showing an example of the main configuration of a detection device, etc. The detection device 510 shown in Fig. 22 is, for example, a device that acquires image data using an image sensor 511, and functions as an imaging device or a ToF sensor. The detection device 510 has the image sensor 511, a signal processing unit 512, and a memory 513. The image sensor 511 supplies image data to the signal processing unit 512 as sensing data.

[0144] The signal processing unit 512 performs signal processing on the image data supplied from the image sensor 511. In doing so, the signal processing unit 512 can appropriately use the memory 513 to store the image data. That is, the signal processing unit 512 can supply the image data to the memory 513 via the bus 521 and store it therein. The signal processing unit 512 can also read out the image data stored in the memory 513 via the bus 522. For example, the signal processing unit 512 can read out and refer to the image data stored in the memory 513 as a past frame, and perform noise reduction processing in the time direction, etc.

[0145] Furthermore, the signal processing unit 512 can supply the signal-processed image data to another information processing device 530 via a transmission path 523. The information processing device 530 can acquire the image data supplied via the transmission path 523.

[0146] Furthermore, the signal processing unit 512 can supply the signal-processed image data to the storage device 540 via a transmission path 524 for storage. Also, another information processing device 530 can read and acquire the image data stored in the storage device 540 via a transmission path 525.

[0147] In such systems, encoding image data can reduce bandwidth and storage capacity during transmission and storage. When this image data may contain reserved words, applying this technology can control the amount of code while preserving the pixel values ​​of some of the sensing data.

[0148] For example, in the signal processing unit 512, the encoding device 100 may be applied as an interface for the bus 521. Also, the decoding device 200 may be applied as an interface for the bus 522. That is, the signal processing unit 512 may encode image data using the encoding device 100 and store the encoded data in the memory 513. Then, the signal processing unit 512 may decode the encoded image data read from the memory 513 using the decoding device 200 to obtain image data.

[0149] Similarly, in the signal processing unit 512, the encoding device 100 may be applied as an interface to the transmission path 523 and the transmission path 524. Furthermore, in the information processing device 530, the decoding device 200 may be applied as an interface to the transmission path 523 and the transmission path 524.

[0150] By doing this, it is possible to reduce the bandwidth used in bus 521 and bus 522, and transmission paths 523 to 525, and the storage capacity used in memory 513 and storage device 540, while suppressing deterioration of reserved words.

[0151] <Application example 2> This technology can also be applied to a polarization camera. Fig. 23 is a block diagram showing an example of the main configuration of a polarization camera. The polarization camera 600 shown in Fig. 23 has a polarization sensor 601 and a signal processing unit 602. The polarization sensor 601 photoelectrically converts incident light at each pixel to generate image data and supplies it to the signal processing unit 602. The signal processing unit 602 performs predetermined signal processing using the image data, deriving and outputting the degree of polarization, polarization phase, average pixel value, etc.

[0152] For example, as shown in A of FIG. 24, the polarization sensor 601 has an on-chip lens 611, a polarizer 612, and an image sensor 613, which are stacked. As shown in the figure, the polarizer 612 is composed of polarizers oriented in four directions: vertical, horizontal, and diagonal. In such a case, pixel values ​​of pixels of the image sensor 613 corresponding to some of the polarizers may become saturated, and light sensitivity may decrease in pixels of the image sensor 613 corresponding to other polarizers. In the example of B of FIG. 24, the pixel value of the image data 621 corresponding to the upper left polarizer of the polarizer 612 is low, and the pixel value corresponding to the lower right polarizer of the polarizer 612 is saturated.

[0153] In such a case, reserved words may be included in the sensing data output by the polarization sensor 601. In this case, by applying the present technology (the encoding device 100 or the decoding device 200) to the signal processing unit 602, it is possible to control the code amount while guaranteeing some pixel values ​​of the sensing data.

[0154] <Application example 3> This technology can be applied not only to iToF but also to dToF systems. It can also be applied to DVS (Dynamic Vision Sensor), which performs asynchronous processing for each pixel, detects changes in the brightness of the object being photographed, and outputs the data.

[0155] Image sensors can develop pixel defects, which are pixels that cannot be captured during manufacturing or use. These pixel defects often appear as black dots with a pixel value of 0 or white dots with a saturated pixel value. When using a general image sensor, it is relatively easy to determine whether a pixel is defective or normal by looking at the difference in pixel values ​​from the surrounding pixels.

[0156] In contrast, in the case of sensors such as dToF, which output pixel values ​​at the timing of receiving infrared light, or sensors such as DVS, which output pixel values ​​only when the brightness value changes, the values ​​of the surrounding pixels cannot be determined to be defective pixels, and it is necessary to separately indicate that the pixel is defective.

[0157] For example, there are cases where the output from a DVS sensor is integrated over a fixed time period and used as image data to understand fluctuations in pixel values. Even if the value is 0 due to a defective pixel, it is difficult to determine from the data alone whether the defective pixel occurred or whether the event never occurred in the first place. When it is necessary to distinguish between these two, a reserved word can be prepared in the output data to record the pixel defect position.

[0158] dToF calculates the depth from the sensor output data based on changes in the received light luminance value, but like DVS, it is difficult to confirm the location of a pixel defect from the data after depth calculation alone. For this reason, there are cases where a reserved word indicating a defective pixel is set to the corresponding pixel value in the detected depth data and output.

[0159] In these cases, by applying this technology, it is possible to control the amount of code while preserving some pixel values ​​(i.e., reserved words) of the sensing data.

[0160] <5. Notes> <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0161] FIG. 25 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0162] In a computer 900 shown in FIG. 25, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0163] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0164] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0165] In a computer configured as above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0166] The program executed by the computer can be applied by recording it on removable media 921 such as package media, for example. In this case, the program can be installed in storage unit 913 via input / output interface 910 by inserting removable media 921 into drive 915.

[0167] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0168] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0169] <Applicable targets of this technology> The present technology can be applied to any configuration, for example, to transmitters and receivers (e.g., television receivers and mobile phones) in satellite broadcasting, cable TV and other wired broadcasting, distribution over the Internet, and distribution to terminals via cellular communication, as well as various electronic devices such as devices that record images on media such as optical disks, magnetic disks, and flash memories, and play images from these storage media (e.g., hard disk recorders and cameras).

[0170] Furthermore, for example, the present technology can also be implemented as a part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0171] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, AV (Audio Visual) equipment, a portable information processing terminal, or an IoT (Internet of Things) device.

[0172] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0173] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications thereof are also arbitrary.

[0174] <Other> The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0175] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0176] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0177] Also, for example, each step of a single flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0178] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0179] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0180] The present technology can also be configured as follows. (1) a data discriminator that discriminates whether each pixel value of the sensing data is a reserved word, which is a predetermined value, and generates a flag map that indicates the positions of reserved word pixels, which are pixels whose pixel values ​​are the reserved words, and non-reserved word data that is composed of pixel values ​​of non-reserved word pixels, which are pixels that are not the reserved word pixels; a flag encoding unit that losslessly encodes the flag map generated by the data discriminator to generate a flag stream; a pixel encoding unit that encodes the non-reserved word data generated by the data discriminator and generates a pixel stream; An information processing device comprising: (2) The image processing device further includes a multiplexing unit that multiplexes the flag stream generated by the flag encoding unit and the pixel stream generated by the pixel encoding unit to generate a sensing stream. The information processing device described in (1). (3) The sensing stream is information with a fixed bit length. (2) An information processing device according to the present invention. (4) The pixel encoding unit generates the pixel stream having a bit length that is a difference between the bit length of the fixed length of the sensing stream and the bit length of the flag stream. (3) An information processing device according to the present invention. (5) the flag encoding unit supplies information about the bit length of the generated flag stream to the pixel encoding unit; The pixel encoding unit determines the bit length of the pixel stream based on the information. (4) An information processing device according to the present invention. (6) The flag encoding unit supplies flag information indicating whether the reserved word pixel is included in the flag map to the pixel encoding unit; The pixel encoding unit determines a bit length of the pixel stream based on the value of the flag information. (5) An information processing device according to (5). (7) The flag encoding unit If the flag information has a value indicating that the reserved word pixel is not included in the flag map, generating the flag stream consisting of only the flag information; If the flag information has a value indicating that the reserved word pixel is included in the flag map, the flag stream is generated, which is composed of the flag information and the flag map. (6) An information processing device according to the present invention. (8) The flag map is fixed-length information. An information processing device according to any one of (1) to (7). (9) The flag map is variable-length information. An information processing device according to any one of (1) to (7). (10) For each pixel value of the sensing data, whether it is a reserved word, which is a predetermined value, is discriminated, and a flag map indicating the positions of reserved word pixels, which are pixels having the reserved word as a pixel value, and non-reserved word data consisting of pixel values ​​of non-reserved word pixels, which are pixels that are not the reserved word pixels, is generated; losslessly encoding the generated flag map to generate a flag stream; The generated non-reserved word data is encoded to generate a pixel stream. Information processing methods.

[0181] (11) a flag decoding unit that losslessly decodes a flag stream included in the sensing stream and generates a flag map that indicates the positions of reserved word pixels, which are pixels having pixel values ​​that are predetermined reserved words; a pixel decoding unit that decodes a pixel stream that is coded data of non-reserved word data composed of pixel values ​​of non-reserved word pixels that are pixels that are not the reserved word pixels and that are included in the sensing stream, and generates a pixel map that includes the non-reserved word data; a data synthesis unit that synthesizes the flag map generated by the flag decoding unit and the pixel map generated by the pixel decoding unit to generate sensing data; An information processing device comprising: (12) The image processing device further includes a demultiplexing unit that demultiplexes the sensing stream and extracts the pixel stream. (11) An information processing device according to (11). (13) The sensing stream is information having a fixed bit length. (12) An information processing device according to (12). (14) The demultiplexing unit extracts the pixel stream having a bit length that is a difference between the fixed bit length of the sensing stream and the bit length of the flag stream. (13) An information processing device according to (13). (15) The flag decoding unit supplies information about a bit length of the flag stream to the demultiplexing unit; The demultiplexing unit extracts the pixel stream from the sensing stream based on the information. (14) An information processing device according to (14). (16) The flag decoding unit supplies flag information indicating whether the reserved word pixel is included in the flag map to the demultiplexing unit; The demultiplexer extracts the pixel stream from the sensing stream based on the value of the flag information. (15) An information processing device according to (15). (17) The flag decoding unit losslessly decoding the flag stream to generate the flag information; If the flag information has a value indicating that the reserved word pixel is not included in the flag map, generating the flag map in which all pixels are not the reserved word pixels; If the flag information has a value indicating that the reserved word pixel is included in the flag map, the flag stream is losslessly decoded to generate the flag map. (16) An information processing device according to (16). (18) The flag map is fixed-length information. An information processing device according to any one of (11) to (17). (19) The flag map is variable-length information. An information processing device according to any one of (11) to (17). (20) losslessly decoding a flag stream included in the sensing stream to generate a flag map indicating the positions of reserved word pixels, which are pixels having pixel values ​​that are predetermined reserved words; decoding a pixel stream that is encoded data of non-reserved word data composed of pixel values ​​of non-reserved word pixels that are pixels that are not the reserved word pixels and that are included in the sensing stream, and generating a pixel map that includes the non-reserved word data; The generated flag map and the generated pixel map are combined to generate sensing data. Information processing methods. [Explanation of symbols]

[0182] 100 encoding device, 101 block division unit, 102 sensing data encoding unit, 103 output unit, 131 data discrimination unit, 132 flag encoding unit, 133 pixel encoding unit, 134 multiplexing unit, 161 reserved word pixel processing unit, 162 6-bit length encoding unit, 163 8-bit length encoding unit, 164 selection unit, 200 decoding device, 201 input unit, 202 sensing data decoding unit, 203 block reconstruction unit, 231 flag decoding unit, 232 demultiplexing unit, 233 pixel decoding unit, 234 data synthesis unit, 510 detection device, 511 image sensor, 512 signal processing unit, 513 memory, 530 information processing device, 540 storage device, 600 polarization camera, 601 polarization sensor 602 Signal Processing Unit, 900 Computer< / itof>

Claims

1. a data discriminator that discriminates whether each pixel value of the sensing data is a reserved word, which is a predetermined value, and generates a flag map that indicates the positions of reserved word pixels, which are pixels having the reserved word as a pixel value, and non-reserved word data that is composed of pixel values ​​of non-reserved word pixels, which are pixels that are not the reserved word pixels; a flag encoding unit that losslessly encodes the flag map generated by the data discriminator to generate a flag stream; a pixel encoding unit that encodes the non-reserved word data generated by the data discriminator and generates a pixel stream whose bit length is the difference between the fixed bit length of the sensing stream and the bit length of the flag stream; a multiplexing unit that multiplexes the flag stream generated by the flag encoding unit and the pixel stream generated by the pixel encoding unit to generate the sensing stream; An information processing device comprising:

2. the flag encoding unit supplies information about the bit length of the generated flag stream to the pixel encoding unit; The pixel encoding unit determines the bit length of the pixel stream based on the information. The information processing device according to claim 1 .

3. the flag encoding unit supplies flag information indicating whether the reserved word pixel is included in the flag map to the pixel encoding unit; The pixel encoding unit determines a bit length of the pixel stream based on the value of the flag information. The information processing device according to claim 2 .

4. The flag encoding unit If the flag information has a value indicating that the reserved word pixel is not included in the flag map, generating the flag stream consisting of only the flag information; If the flag information has a value indicating that the reserved word pixel is included in the flag map, the flag stream is generated, which is composed of the flag information and the flag map. The information processing device according to claim 3 .

5. The flag map is fixed-length information. The information processing device according to claim 1 .

6. The flag map is variable-length information. The information processing device according to claim 1 .

7. for each pixel value of the sensing data, discriminating whether it is a reserved word, which is a predetermined value, and generating a flag map indicating the positions of reserved word pixels, which are pixels having the reserved word as a pixel value, and non-reserved word data consisting of pixel values ​​of non-reserved word pixels, which are pixels that are not the reserved word pixels; losslessly encoding the generated flag map to generate a flag stream; encoding the generated non-reserved word data to generate a pixel stream whose bit length is the difference between the fixed bit length of the sensing stream and the bit length of the flag stream; The generated flag stream and the generated pixel stream are multiplexed to generate the sensing stream. Information processing methods.

8. a flag decoding unit that losslessly decodes a flag stream included in the sensing stream and generates a flag map that indicates the positions of reserved word pixels, which are pixels having pixel values ​​that are predetermined reserved words; a demultiplexing unit that demultiplexes the sensing stream having a fixed bit length and extracts a pixel stream having a bit length that is a difference between the fixed bit length of the sensing stream and the bit length of the flag stream; a pixel decoding unit that decodes the pixel stream, which is encoded data of non-reserved word data composed of pixel values ​​of non-reserved word pixels that are not the reserved word pixels extracted by the demultiplexing unit, and generates a pixel map including the non-reserved word data; a data synthesis unit that synthesizes the flag map generated by the flag decoding unit and the pixel map generated by the pixel decoding unit to generate sensing data; An information processing device comprising:

9. the flag decoding unit supplies information about the bit length of the flag stream to the demultiplexing unit; The demultiplexing unit extracts the pixel stream from the sensing stream based on the information. The information processing device according to claim 8 .

10. the flag decoding unit supplies flag information indicating whether the reserved word pixel is included in the flag map to the demultiplexing unit; The demultiplexer extracts the pixel stream from the sensing stream based on the value of the flag information. The information processing device according to claim 9 .

11. The flag decoding unit losslessly decoding the flag stream to generate the flag information; If the flag information has a value indicating that the reserved word pixel is not included in the flag map, generating the flag map in which all pixels are not the reserved word pixels; If the flag information has a value indicating that the reserved word pixel is included in the flag map, the flag stream is losslessly decoded to generate the flag map. The information processing device according to claim 10.

12. The flag map is fixed-length information. The information processing device according to claim 8 .

13. The flag map is variable-length information. The information processing device according to claim 8 .

14. losslessly decoding a flag stream included in the sensing stream, and generating a flag map indicating the positions of reserved word pixels, which are pixels having pixel values ​​that are predetermined reserved words; demultiplexing the sensing stream having a fixed bit length, and extracting a pixel stream having a bit length equal to the difference between the fixed bit length of the sensing stream and the bit length of the flag stream; decoding the pixel stream, which is encoded data of non-reserved word data composed of pixel values ​​of non-reserved word pixels that are pixels that are not the reserved word pixels extracted from the sensing stream, to generate a pixel map including the non-reserved word data; The generated flag map and the generated pixel map are combined to generate sensing data. Information processing methods.

Images