Information processing device and data structure

Abstract

An information processing device according to one aspect of the present technology is provided with a first generation unit, a second generation unit, and an output unit. The first generation unit generates insertion data obtained by inserting, between image data of a first image, image data of at least one second image different from the first image. The second generation unit generates metadata including a insertion position of the image data of the second image in the insertion data. The output unit outputs an image frame including the insertion data and the metadata.

Description

Information processing device and data structure

[0001] The present technology relates to an information processing device and a data structure used for transmitting images.

[0002] Conventionally, systems that transmit images and perform processing such as display have been known. For example, systems that combine an inspection camera and a display monitor are widely used for various inspections. Patent Document 1 describes a technology that generates a full image and a partial image of an object captured by a visual sensor, processes the images, and simultaneously displays them on a display device.

[0003] Japanese Patent Application Laid-Open No. 2017-151813

[0004] When transmitting multiple images, such as a full image and partial images, one method would be to transmit each image through a separate output interface (transmission system), but this method requires an increased number of output interfaces. Furthermore, the image transmission timing for each output interface is not necessarily simultaneous. Therefore, the destination device must be provided with a sufficient buffer to synchronize the processing timing.

[0005] As such, adding output interfaces and buffers can increase the cost of implementing a system, so there is a need for technology that can simplify the configuration of a system that transmits multiple images.

[0006] In view of the above circumstances, an object of the present technology is to provide an information processing device and a data structure that can simplify the configuration of a system that transmits multiple images.

[0007] To achieve the above object, an information processing device according to one embodiment of the present technology includes a first generating unit, a second generating unit, and an output unit. The first generating unit generates insertion data in which image data of at least one second image different from the first image is inserted between image data of the first image. The second generating unit generates metadata including an insertion position of the image data of the second image in the insertion data. The output unit outputs an image frame including the insertion data and the metadata.

[0008] This information processing device outputs an image frame including insertion data in which image data of at least one second image is inserted between image data of first images, and metadata including the insertion position of the image data of the second images. This makes it possible to transmit data of multiple images in a single image frame. Furthermore, by referencing the metadata, it becomes possible to separate each image in parallel from the image frame. As a result, it becomes possible to simplify the configuration of a system for transmitting multiple images.

[0009] The image data of at least one of the first image and the second image may include image data obtained by thinning out pixels of an original image.

[0010] The image data of the first image and the image data of the second image may each be configured as a pixel block, and the insertion data may be data in which the pixel blocks are arranged in an image data area of ​​the image frame.

[0011] The insert data may be data in a matrix format, in which case the second generator may generate the metadata for each row of the insert data.

[0012] The first image may be a whole image representing the entire input image. In this case, the second image may be a partial image representing a part of the input image. The first generation unit may generate the insertion data by inserting image data of the partial image into image data of the whole image at a position corresponding to the partial image.

[0013] The first generation unit may determine whether or not there is an abnormality in the input image in units of processing blocks of a predetermined size, and generate image data of the partial image from image data of the input image in an abnormal block, which is the processing block determined to have the abnormality.

[0014] The first generating unit may generate the image data of the partial image by using all pixels of the input image in the abnormal block.

[0015] The first generating unit may generate the image data of the entire image by thinning out pixels of the input image in the abnormal block.

[0016] The first generating unit may delete a portion of the image data of the entire image when inserting the image data of the partial image causes the amount of data in the image data area of ​​the image frame to exceed a predetermined value.

[0017] The first generation unit may increase the amount of data in the image data area of ​​the image frame by the amount that exceeds a predetermined value if inserting image data of the partial image causes the amount of data in the image data area of ​​the image frame to exceed a predetermined value.

[0018] The first generating unit may generate the image data of the entire image and the image data of the partial image by thinning out pixels of the input image.

[0019] The first generating unit may thin out pixels of the input image so that the total data amount of the image data of the entire image and the image data of the partial image in the abnormal block falls within the data amount of the abnormal block.

[0020] The first generating unit may generate the image data of the entire image and the image data of the partial image by thinning out the pixels of the input image in the abnormal block by half.

[0021] The metadata may include a result of determining whether or not an abnormality exists.

[0022] The metadata may include the type of the anomaly.

[0023] The input image may be an image captured by a solid-state image sensor, in which case the information processing device may be configured as a sensor unit including the solid-state image sensor.

[0024] According to one aspect of the present technology, an information processing device includes a receiving unit and a separating unit. The receiving unit receives image frames including insertion data in which image data of at least one second image other than the first image is inserted between image data of the first image, and metadata including an insertion position of the image data of the second image in the insertion data. The separating unit separates the image data of the first image and the image data of the second image from the insertion data based on the metadata.

[0025] A data structure according to one embodiment of the present technology includes insertion data in which image data of at least one second image other than the first image is inserted between image data of the first image, and metadata including the insertion position of the image data of the second image in the insertion data, and is used in transmission processing of the first image and the second image.

[0026] FIG. 1 is a schematic diagram showing a configuration example of an inspection system including an imaging sensor unit according to a first embodiment of the present technology; FIG. 1 is a schematic diagram showing examples of an imaged image, a full image, and a partial image; FIG. 2 is a schematic diagram showing an example of an image frame; FIG. 3 is a schematic diagram showing an example of metadata of an image frame; FIG. 4 is a flowchart showing an operation example of an imaging controller; FIG. 5 is a schematic diagram showing an operation example of an imaging controller; FIG. 6 is a schematic diagram showing an example of image thinning processing; FIG. 7 is a schematic diagram showing another example of an image frame; FIG. 8 is a flowchart showing an operation example of a display controller; FIG. 9 is a schematic diagram showing an example of a display image displayed on a display unit; FIG. 10 is a schematic diagram showing a configuration example of an inspection system given as a comparative example; FIG. 11 is a schematic diagram showing a configuration example of an image transmission system according to a second embodiment;

[0027] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0028] 1 is a schematic diagram showing an example of the configuration of an inspection system including an image sensor unit according to a first embodiment of the present technology. The inspection system 100 has an image sensor unit 10 and a display unit 30, and is a system that inspects an object by displaying an image captured by the image sensor unit 10 to a user on the display unit 30.

[0029] In the inspection system 100, an entire image 2 and a plurality of partial images 3 cut out from a captured image 1 captured by the imaging sensor 11 of the imaging sensor unit 10 are displayed to the user. Here, the entire image 2 is an image that represents the entire captured image 1. The partial images 3 are images that represent a part of the captured image. In this embodiment, the captured image 1 corresponds to the input image, the entire image 2 corresponds to the first image, and the partial images 3 correspond to the second image.

[0030] For example, when inspecting an object in real time, a method is used in which each part of the object is photographed in turn while changing the photographing range of the object, and the photographed image 1 is read, and at the same time, whether or not there is an abnormality in the read part is determined. In this method, by displaying a partial image 3 of the part of the photographed image 1 that is determined to be abnormal, in addition to the entire image 2 of the photographed image 1, it is possible to check the abnormal part in detail.

[0031] The inspection system 100 is a system that can be applied to such a method. The image sensor unit 10 is configured to perform image processing such as anomaly detection therein, generate an entire image 2 and a partial image 3, and output the data to a display unit at a subsequent stage.

[0032] [Image Sensor Unit] As shown in FIG. 1, the image sensor unit 10 includes an image sensor 11 , a storage unit 12 , and an image controller 13 .

[0033] The imaging sensor 11 is a solid-state imaging element that captures an image of an object and outputs a captured image 1 of the object. Therefore, the captured image 1 can also be considered raw data output from the imaging sensor 11. As the imaging sensor 11, for example, a CMOS (Complementary Metal-Oxide Semiconductor) image sensor (CIS) is used. The imaging sensor unit 10 itself can also be called a CIS. Note that the type of imaging sensor 11 is not limited, and for example, a CCD (Charge Coupled Device) image sensor or the like may be used.

[0034] The storage unit 12 is a non-volatile storage device, such as a solid state drive (SSD) or a hard disk drive (HDD). The storage unit 12 stores a control program for operating the image sensor unit 10. The storage unit 12 also stores setting parameters used for processing, such as generating the entire image 2 and the partial images 3. In this embodiment, the storage unit 12 corresponds to a computer-readable recording medium on which a program is recorded. The control program corresponds to the program recorded on the recording medium.

[0035] The imaging controller 13 is a computing device that controls the operation of the imaging sensor unit 10. The imaging controller 13 has hardware components necessary for a computer, such as a CPU and memory (RAM, ROM). The CPU loads a control program stored in the storage unit 12 into the RAM and executes it, thereby performing various processes. In this embodiment, the imaging controller 13 corresponds to an information processing device that outputs image frames, and is configured as a sensor unit including a solid-state imaging element.

[0036] The imaging controller 13 may be, for example, a programmable logic device (PLD) such as a field programmable gate array (FPGA), or another device such as an application specific integrated circuit (ASIC). Alternatively, the imaging controller 13 may be configured using any other logic circuit.

[0037] In this embodiment, the CPU of the imaging controller 13 executes a program (control program) according to this embodiment, thereby realizing functional blocks including an insertion data generation unit 14, a metadata generation unit 15, and a data output unit 16. These functional blocks then execute the information processing method according to this embodiment. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

[0038] The insert data generation unit 14 generates insert data 4 by inserting image data of at least one partial image 3 separate from the entire image 2 into image data of the entire image 2. Here, image data refers to data representing a part or all of an image, and is data in which the values ​​of multiple pixels constituting the image are arranged. Therefore, the insert data 4 is data in which the array of pixel values ​​of the partial image 3 is inserted into the array of pixel values ​​of the entire image 2. The specific configuration of the insert data 4 will be described later with reference to Figures 3, 8, 9, etc. In this embodiment, the insert data generation unit 14 corresponds to a first generation unit.

[0039] The insert data generating unit 14 includes an abnormality detecting unit 17 , a flag determining unit 18 , an entire image generating unit 19 , a partial image generating unit 20 , and a data arranging unit 21 .

[0040] The anomaly detection unit 17 reads the image data of the captured image 1 (captured image data) output from the imaging sensor 11, and performs an anomaly detection process on the captured image 1. This anomaly detection process is performed in units of processing units, which will be described later. The anomaly detection unit 17 also adds an anomaly flag indicating the presence or absence of an anomaly to the image data that has undergone the anomaly detection process, and outputs the image data. The anomaly detection unit 17 also generates information indicating the type of anomaly, and a count value for the processing block. The count value for the processing block is, for example, data indicating the position within the captured image 1 of the area where the most recent anomaly detection process was performed.

[0041] The flag determination unit 18 reads the image data of the captured image 1 that has been subjected to the abnormality detection process from the abnormality detection unit 17, and determines whether or not there is an abnormality based on the abnormality flag set on the image data. The flag determination unit 18 also outputs image data that has been determined to have no abnormality only to the entire image generation unit 19, and outputs image data that has been determined to have an abnormality to both the entire image generation unit 19 and the partial image generation unit 20.

[0042] The overall image generating unit 19 generates image data that will become the overall image 2 from the image data of the captured image 1 in accordance with the setting parameters read from the storage unit 12. Specifically, image data that represents a portion of the overall image 2 is generated using all or a portion of the image data read in processing block units.

[0043] The partial image generating unit 20 generates image data that will become the partial image 3 from the image data of the captured image 1 in accordance with the setting parameters read from the storage unit 12. Specifically, image data that represents a part of the partial image 3 is generated using all or part of the image data read in processing block units.

[0044] The setting parameters include the thinning ratio of the image data of the captured image 1. The thinning ratio used by the entire image generating section 19 and the thinning ratio used by the partial image generating section 20 are parameters that can be set independently of each other.

[0045] The data arrangement unit 21 arranges the image data of the entire image 2 generated by the entire image generation unit 19 and the image data of the partial image 3 generated by the partial image generation unit 20 to generate the insert data 4. In this embodiment, the image data of the entire image 2 and the partial image 3 generated from the same processing block are arranged so that they are adjacent to each other. More specifically, the insert data 4 is data in which the image data of the entire image 2 and the partial image 3 are arranged in order within an image data area set in an image frame 6, which will be described later. This point will be described in detail later with reference to FIG. 3.

[0046] The metadata generation unit 15 generates metadata 5 corresponding to the insert data 4. The metadata 5 includes various information other than pixel values ​​related to the insert data. In the present disclosure, the metadata generation unit 15 generates metadata 5 including the insertion position of image data of the partial image 3 in the insert data 4. In other words, the metadata 5 is data that records which image data in the insert data 4 is data of the partial image. In this embodiment, the metadata generation unit 15 corresponds to a second generation unit.

[0047] The metadata generating unit 15 includes a determination information generating unit 22 , a type information generating unit 23 , and a position information generating unit 24 .

[0048] The determination information generating unit 22 reads the abnormality flag from the abnormality detecting unit 17 and generates determination information indicating the determination result of whether or not an abnormality exists. The abnormality determination information is information in a format defined in the metadata 5 that indicates the presence or absence of an abnormality for each processing block.

[0049] The type information generator 23 reads information indicating the type of abnormality from the abnormality detector 17 and generates type information indicating the type of abnormality. The abnormality type information is information that indicates the type of abnormality for each processing block in a format defined in the metadata 5.

[0050] The position information generation unit 24 reads the count value of the processing block from the abnormality detection unit 17 and the size of the processing block from the storage unit. Based on this information, the position information generation unit 24 generates position information that indicates the insertion position of the image data of the partial image 3 in the insertion data 4. The position information records, for example, the pixel position that indicates the range of the image data of the partial image 3.

[0051] The data output unit 16 outputs an image frame 6 including the insert data 4 and the metadata 5. Specifically, the data output unit 16 integrates the insert data 4 output from the data arrangement unit 21 and the metadata 5 including the information output from the metadata generation unit 15 (anomaly determination information, anomaly type information, and insertion position information) into one image frame 6. At this time, the image frame 6 is configured in accordance with an output data format for transmitting an image.

[0052] In this way, the image frame 6 includes insertion data 4 in which image data of at least one partial image 3 other than the entire image 2 is inserted between image data of the entire image 2, and metadata 5 including the insertion position of the image data of the partial image 3 in the insertion data 4, and is data having a data structure used in the transmission process of the entire image 2 and the partial image 3. In this embodiment, the data output unit 16 corresponds to the output unit. The data structure of the image frame 6 also corresponds to the data structure used in the transmission process of the first image and the second image.

[0053] As shown in FIG. 1 , the image frame 6 is transmitted to the downstream display unit 30 via a single output interface 25 (output IF). Here, the output interface 25 is a transmission system used for transmitting data. For example, a transmission line such as a coaxial cable functions as the output interface 25. Alternatively, a transmission channel in a communication module can also be called the output interface 25. In this way, the inspection system 100 is a system in which the image sensor unit 10 and the display unit 30 are connected via a single output interface 25.

[0054] [Display Unit] The display unit 30 receives the image frames 6 output from the imaging sensor unit 10, and generates and displays the entire image 2 and the partial image 3 from the image frames 6. As shown in FIG. 1 , the display unit 30 has a display device 31, a storage unit 32, and a display controller 33.

[0055] The display device 31 is a device that displays an image of an object, inspection results, etc. in the inspection system 100. As the display device 31, any display element such as a liquid crystal display or an organic EL display can be used.

[0056] The storage unit 32 is a non-volatile storage device, such as an SSD or HDD. The storage unit 32 stores a program (receiving program) that receives image frames 6 having the data structure according to this embodiment and generates the entire image 2 and the partial image 3. The receiving program is configured as, for example, an application program for inspecting an object. Note that the receiving program may be configured separately from the application program.

[0057] The display controller 33 is a computing device that controls the operation of the display unit 30. The display controller 33 has hardware components necessary for a computer, such as a CPU and memory (RAM, ROM). The CPU loads a reception program stored in the storage unit 32 into the RAM and executes it, thereby performing various processes. In this embodiment, the display controller 33 corresponds to an information processing device that receives image frames.

[0058] The display controller 33 may be a PLD such as an FPGA, or another device such as an ASIC. Alternatively, the imaging controller 13 may be configured using any other logic circuit.

[0059] In this embodiment, the CPU of the display controller 33 executes the reception program to realize functional blocks including a data receiving unit 34, an image separating unit 35, and a display image generating unit 36. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

[0060] The data receiving unit 34 receives an image frame 6 including insertion data 4 in which image data of at least one partial image 3 separate from the entire image 2 is inserted between image data of the entire image 2, and metadata 5 including the insertion position of the image data of the partial image 3 in the insertion data 4. In addition, the data receiving unit 34 receives various types of data transmitted from devices external to the display controller 33. In this embodiment, the data receiving unit 34 corresponds to a receiving unit.

[0061] The image separation unit 35 separates the image data of the entire image 2 and the image data of the partial image 3 from the insertion data 4 based on the metadata 5. Specifically, the image separation unit 35 selects the image data of the entire image 2 and the image data of the partial image 3 by referring to the position information recorded in the metadata 5. In this embodiment, the image separation unit 35 corresponds to a separation unit.

[0062] The display image generating unit 36 ​​generates the entire image 2 and the partial image 3 using the image data of the entire image 2 and the partial image 3. The display image generating unit 36 ​​also generates a display image including the entire image 2 and the partial image 3. Here, the display image is an image output to the display device 31, and is, for example, an image that constitutes a GUI (Graphical User Interface) such as an examination screen. In the display image, the entire image 2 and the partial image 3 are displayed simultaneously. The display image will be described later with reference to FIG. 11 etc.

[0063] [Captured Image, Whole Image, Partial Image] Fig. 2 is a schematic diagram showing an example of a captured image, a whole image, and a partial image. Fig. 2A shows a captured image 1 output from the imaging sensor 11. In this embodiment, the captured image 1 is treated as one frame, and the frame is divided into a plurality of regions (processing blocks 40) for sequential image processing. Specifically, the anomaly detection unit 17 of the insertion data generation unit 14 performs anomaly detection processing on the captured image 1 in units of processing blocks 40.

[0064] 2A, processing blocks 40 applied to the captured image 1 are schematically illustrated as rectangular regions. The processing blocks 40 are processing unit regions for image processing, and are configured as pixel blocks of a predetermined size. A block number (region number) is assigned to each processing block 40. Hereinafter, image data of the captured image 1, which is made up of processing blocks 40 as a unit, will be referred to as image data 41a. Therefore, the captured image 1 will be data in which multiple image data 41a are arranged in a grid pattern.

[0065] Here, the size of the processing block 40 is m pixels x n rows. For example, a pixel block of 10 pixels x 10 rows is used as the processing block 40. Of course, the size of the processing block 40 is not limited to this and can be set appropriately depending on the type of object, etc.

[0066] In the anomaly detection process, an abnormal portion 38 of the object is detected for each processing block 40. In the example shown in FIG. 2A, a linear abnormal portion 38 (e.g., a scratch or crack) on the surface of the object is schematically illustrated. Hereinafter, a processing block 40 determined to have an abnormality will be referred to as an abnormal block 40e. In FIG. 2A, the processing blocks with block numbers 3, 9, 10, and 11 are abnormal blocks 40e in which abnormal portions 38 are detected. Furthermore, an abnormality flag indicating the presence of an abnormality is added to the image data of the abnormal block 40e.

[0067] Any algorithm can be used as the detection algorithm for detecting abnormalities. For example, edge detection based on brightness or detection based on comparison with an abnormal pattern (pattern matching, etc.) can be used. In addition, any algorithm that can properly perform the desired abnormality detection may be used depending on the type of object, the inspection items, etc.

[0068] 2B schematically illustrates an overall image 2 representing the entire captured image 1. The overall image 2 is generated by outputting image data 41a of the entire region of the captured image 1 in units of processing blocks 40. Hereinafter, the image data of the overall image 2 generated from the image data 41a will be referred to as image data 41b.

[0069] For example, for the abnormal block 40e, data obtained by thinning out the image data 41a of the captured image 1 is used as the image data 41b. For other processing blocks 40, the image data 41a of the captured image 1 is used as the image data 41b as is. Note that there are also cases where data obtained by thinning out the image data 41a is used for processing blocks 40 that are not abnormal. This point will be described later.

[0070] 2C schematically illustrates a partial image 3 representing an abnormal region 38 in the captured image 1. The partial image 3 is generated by outputting image data 41a of an abnormal block 40e among the multiple processing blocks 40. Hereinafter, the image data of the partial image 3 generated from the image data 41a will be referred to as image data 41c.

[0071] For example, for each abnormal block 40e, the image data 41c may be data that uses the image data 41a of the captured image 1 as is, or data that thins out the image data 41a of the captured image 1. In this way, the partial image 3 is output according to the abnormal region 38, etc., and becomes an image in which the image data 41c are arranged in a mosaic pattern.

[0072] Note that one partial image 3 may be generated for one captured image 1. In this case, the partial image 3 is an image that represents abnormal regions 38 in the entire region of the captured image 1. Also, a plurality of partial images 3 may be generated for one captured image 1. For example, in the example shown in FIG. 2A , there are abnormal regions 38 spaced apart from each other on the upper and lower sides of the figure. Partial images 3 showing these abnormal regions 38 may be generated respectively.

[0073] The partial image 3 may also be obtained by processing the image data 41a of the entire image 2. For example, processing such as adding predetermined information, rotating, or trimming may be performed. Here, processing to add information includes, for example, processing to emphasize edges or adding text data. In addition, any image processing may be performed when constructing the partial image 3.

[0074] In this way, in the insertion data generation unit 14, the abnormality detection unit 17 determines whether or not there is an abnormality in the captured image 1 for each processing block 40 of a predetermined size, and the partial image generation unit 20 generates image data 41c of the partial image 3 from the image data 41a of the captured image 1 in the abnormal block 40e. This makes it possible to generate the partial image 3 that displays the abnormal area 38 in detail, thereby fully assisting the user in checking the abnormal area 38, etc.

[0075] [Image Frame] Fig. 3 is a schematic diagram showing an example of an image frame. Fig. 3 also shows a schematic diagram of the structure of an image frame 6 generated by the data output unit 16. The image frame 6 is configured, for example, based on the data size of the captured image 1. In the example shown in Fig. 3, for a processing block 40 determined to be abnormal (abnormal block 40e), a thinned-out entire image 2 (image data 41b) and an unthinned partial image 3 (image data 41c) are output.

[0076] In the image frame 6, the area shown on the left side of the drawing is a metadata area 45 for storing metadata 5, and the area shown on the right side of the drawing is an image data area 46 for storing image data. The image data area 46 stores insertion data 4 in which image data 41b of the entire image 2 and image data 41c of the partial image 3 are inserted.

[0077] In this embodiment, the image data 41b of the entire image 2 and the image data 41c of the partial image 3 are each configured as pixel blocks. That is, the image data 41b and the image data 41c are data in which pixel values ​​are arranged in a grid pattern. As shown in Figure 3, the insertion data 4 is data in which these pixel blocks are arranged in the image data area 46 of the image frame 6.

[0078] By treating the contents of the insertion data 4 as pixel blocks in this way, it becomes possible to easily perform, for example, the process of inserting image data 41c of partial image 3 or the process of separating image data 41c of partial image 3. The sizes of the pixel blocks of image data 41b and image data 41c are determined by the size of processing block 40 and a thinning ratio, which will be described later.

[0079] 3, a region determined to be an abnormal block 40e in the captured image 1 is schematically illustrated by a thick dotted line. As described with reference to FIG. 2 etc., for this region, image data 41b of an entire image 2 is generated by the entire image generating unit 19, and image data 41c of a partial image 3 is generated by the partial image generating unit 20.

[0080] Furthermore, the data arrangement unit 21 arranges the image data 41b of the entire image 2 and the image data 41c of the partial image 3 generated for the abnormal block 40e adjacent to each other. Therefore, the image data 41c of the partial image 3 is recorded consecutively with the image data 41b of the entire image 2 corresponding to the image data 41c. In this way, the data arrangement unit 21 generates the insertion data 4 by inserting the image data 41c of the partial image 3 between the image data 41b of the entire image 2 and at a position corresponding to the partial image 3.

[0081] As a result, the insertion data 4 becomes data that alternately outputs two types of image data 41b and 41c, the entire image 2 and the partial image 3. This makes it possible to output the entire image 2 and the partial image 3 as a single image frame 6, and control them with a single output interface 25. As a result, it is possible to reduce the number of output interfaces 25 compared to, for example, a case where the entire image 2 and the partial image 3 are output using separate output interfaces. It is also possible to eliminate unused, wasted data bandwidth (such as the time the output interface 25 is waiting).

[0082] Furthermore, in the insertion data 4, the corresponding image data 41b and image data 41c are arranged adjacent to each other. Therefore, the image data 41c of the partial image 3 is output at the position where it should be displayed when the partial image 3 is reconstructed. This makes it possible to generate the partial image 3 at a later stage simply by arranging the image data 41c in the appropriate positions in order. In other words, there is no need to read all the image data 41c and then synthesize the partial image 3 again. Because there is no need to synthesize the partial image 3 in this way, it is possible to reduce the storage capacity of the buffer, etc., required for image processing at a later stage.

[0083] [Pixel thinning process] In this embodiment, the image data 41b and 41c of at least one of the entire image 2 and the partial image 3 includes image data in which pixels of the original image (captured image 1) have been thinned out. By thinning out the image data 41b and 41c in this way, it is possible to make the amount of data of, for example, image frame 6 the same as that of captured image 1. This makes it possible to transmit the data of the entire image 2 and the partial image 3 while maintaining the frame rate.

[0084] 3, the partial image generating unit 20 generates image data 41c of the partial image 3 using all pixels of the captured image 1 in the abnormal block 40e. That is, for the partial image 3, the image data 41c is generated without thinning out any pixels from the image data 41a of the captured image 1. This makes it possible to generate the partial image 3 that displays the abnormal region 38 in detail.

[0085] 3, the entire image generating unit 19 thins out pixels of the captured image 1 in the abnormal block 40e to generate image data 41b of the entire image 2. That is, for the entire image 2 of the abnormal block 40e, image data 41b is generated by thinning out pixels from the image data 41a of the captured image 1. Here, the rate at which pixels are thinned out (thinning ratio) is set to 1 / 2.

[0086] 3, image data 41b of the entire image 2 other than the abnormal block 40e is thinned out so as not to exceed the data capacity of the image data area 46. Hereinafter, the processing block 40 located in the Xth row from the top and the Yth column from the left in the figure will be referred to as processing block (X, Y). For example, processing block (2, 1) is the abnormal block 40e. Therefore, image data 41b of the entire image 2 for processing block (2, 1) is thinned out by half. Furthermore, image data 41c of the partial image 3 for processing block (2, 1) is not thinned out (thinning ratio is 1).

[0087] Even if image data 41b of entire image 2 is thinned out for processing block (2,1) in this way, by inserting image data 41c of partial image 3, the amount of data in each row increases by half the amount of data in the processing block. To offset this increased amount of data, image data 41b of entire image 2 is thinned out for processing block (2,2) in Figure 3. As a result, the amount of data in the second row ultimately falls within the amount of data in the original image data area 46 (the amount of data in captured image 1).

[0088] In this way, the image data 41b of the corresponding whole image 2 is thinned out depending on the amount of partial image 3 generated, and the total amount of data in the image frame 6 is maintained at the same level as the data amount of the original captured image 1. This maintains the frame rate at the same level as when only the captured image 1 is transmitted.

[0089] The thinning ratio value is variable to accommodate user requests. For example, if the image quality of the entire image 2 can be low, the thinning ratio for the entire image 2 can be set to a value lower than 1 / 2 (e.g., 1 / 3 or 1 / 4). If the image quality of the partial image 3 can be low, the thinning ratio for the partial image 3 can be set to a value lower than 1 (e.g., 1 / 2) (see FIG. 8). Alternatively, the thinning ratio value may be set so that the user can input any desired value, or may be set according to the inspection mode.

[0090] [Metadata] As described above, the insert data 4 is data in which pixel blocks are arranged, and is data in a matrix format in which pixel values ​​are arranged in a grid pattern. The metadata generator 15 generates metadata 5 for each row of the insert data 4. In Figure 3, the metadata 5 recorded for each line (row) in the metadata area 45 of the image frame 6 is schematically illustrated by the thin dotted line area.

[0091] For example, the first line of the image frame 6 is the line where the frame starts, and the last line of the image frame 6 is the line where the frame ends. The image frame 6 is transmitted, for example, line by line, from the first line to the last line. By inserting the metadata 5 indicating the insertion position of the partial image 3 for each line, it becomes possible to smoothly separate the image data 41c of the entire image 2 from the image data 41b of the partial image 3 each time each line of the image frame 6 is received.

[0092] In order for the user to select the partial image 3, it is necessary for the user to present the position of the image data 41c of the partial image 3. Therefore, as will be described below, the metadata 5 records whether or not an abnormality has been detected and the position of the image data 41c of the partial image 3 where the abnormality occurred.

[0093] 4A and 4B are schematic diagrams showing an example of metadata for an image frame. One line of data for an image frame 6 is shown in each of FIGS. 4A and 4B. The remaining data, excluding the first flag on the left side of the figure, the last flag on the right side of the figure, and the image data immediately before the last flag, constitutes metadata 5.

[0094] As shown in FIG. 4A , determination information 47 indicating whether or not an abnormality has been detected is stored at the beginning of the metadata 5. That is, the metadata 5 includes the determination result of whether or not an abnormality has been detected. The determination information 47 is generated by the determination information generation unit 22 of the metadata generation unit 15. For example, if the determination information indicates "abnormality present," the row contains partial image 3, and therefore, position information 48 becomes valid. Note that if the determination information indicates "abnormality absent," the row does not contain partial image 3. Therefore, there is no need to consider the position information 48.

[0095] In the metadata 5, position information 48 is stored following the determination information 47. The position information 48 is information indicating the insertion position of the image data 41c of the partial image 3 in each row. The position information 48 is generated by the position information generation unit 24 of the metadata generation unit 15. In the example shown in FIG. 4A , the pixel position where the image data 41c of the first partial image 3 starts (start pixel position of partial image #1) and the pixel position where the image data 41c of the first partial image 3 ends (end pixel position of partial image #1) are recorded.

[0096] Here, the pixel position refers to, for example, the pixel position in the column direction in the image data area of ​​each row (which can also be considered as the pixel position within the row). The start pixel position and end pixel position of the n-th image data 41c are recorded as the position information 48, and are recorded for each detected abnormal block 40e. The position information 48 is calculated from the size of the processing block 40 (processing unit area) and the thinning ratio.

[0097] In addition, other metadata 49 is stored in the metadata 5 following the position information 48. The other metadata 49 records, for example, information that is generally used in the image frame 6 (such as the line number of each line). In addition, information such as the thinning rate used in the image data 41c of the entire image 2 may also be stored.

[0098] FIG. 4B is a schematic diagram showing another example of the metadata 5. In the metadata 5 shown in FIG. 4B, type information 50 indicating the type of abnormality is stored following the determination information 47. That is, the metadata includes the type of abnormality. The type information 50 is generated by the type information generation unit 23 of the metadata generation unit 15. As described above, when a partial image 3 is generated, it means that an abnormality has been detected. Therefore, by including the type of abnormality in the metadata 5, it is possible to improve usability.

[0099] In the type information 50, the type of anomaly is recorded as a number. For example, if the anomaly is a "crack", "0" is recorded, if the anomaly is a "distortion", "1" is recorded, and if the anomaly is a "breakage", "2" is recorded. Of course, the correspondence between the type of anomaly and the number is not limited to this, and other types of anomalies may be recorded so as to be distinguishable. Furthermore, the degree of anomaly, etc. may be recorded as metadata 5.

[0100] Returning to Fig. 3 , a case where image data 41b of the entire image 2 is deleted will be described. For example, in the second row of Fig. 3 , the excess data amount can be avoided by thinning out the image data 41b of the entire image 2 for the abnormal block 40e (processing block (2,1)) and the processing block (2,2) immediately thereafter. However, with the method shown in Fig. 3 , depending on the number of abnormal blocks 40e that occur, it may not be possible to avoid the excess data amount simply by thinning out the image data 41b of the entire image 2.

[0101] 3, the processing block (5,3) at the rearmost (right side) is the abnormal block 40e, and in the sixth row, the processing block (6,1) at the frontmost (left side) is the abnormal block 40e. In this case, image data 41b (thinning ratio 1 / 2) of the entire image 2 of the processing block (5,3) is placed in the first half of the area of ​​the processing block (5,3), and half of the image data 41c of the partial image 3 of the processing block (5,3) is placed in the second half of the area.

[0102] At the beginning of the sixth row, the remaining half of the image data 41c of partial image 3 of processing block (5,3) that did not fit into row 5 is placed. Next, image data 41c of partial image 3 of processing block (6,1) is placed, followed by image data 41b (thinning ratio 1 / 2) of whole image 2 of processing block (6,2).

[0103] Note that image data 41b of whole image 2 of processing block (6,1) is not generated because the amount of data would be excessive. Alternatively, image data 41b of whole image 2 of processing block (6,2) may not be generated. Furthermore, for processing blocks 40 in which image data 41b of whole image 2 does not exist, the number or the like may be recorded as metadata 5.

[0104] 3, if inserting image data 41c of partial image 3 causes the amount of data in the image data area 46 of image frame 6 to exceed a predetermined value, part of the image data 41b of entire image 2 is deleted. In this case, for example, part of entire image 2 may become blank. On the other hand, when inspecting an object, it is not always necessary to display entire image 2 in detail. Therefore, if blank areas in entire image 2 do not pose a problem, deleting part of entire image 2 to keep the amount of data within a predetermined value makes it possible to avoid a decrease in frame rate, etc.

[0105] [Operation of Imaging Controller] Fig. 5 is a flowchart showing an example of the operation of the imaging controller. Fig. 6 is a schematic diagram showing an example of the operation of the imaging controller. Below, a basic process for generating the image frame 6 performed by the imaging controller 13 will be described. This process is a process showing an example of a method for producing data (image frame 6) having a data structure according to this embodiment.

[0106] 5 is a loop process that is repeatedly executed for each processing block 40 on the captured image 1 output from the image sensor 11. First, the anomaly detection unit 17 executes anomaly detection processing for the image data 41a of the captured image 1 (step 101). Here, various image processes are executed to detect anomalies in the image data 41a, and a flag indicating the presence or absence of an anomaly in the processing block 40, information on the type of anomaly, and a count value (block number) of the processing block are generated.

[0107] Next, the flag determination unit 18 determines whether or not there is an abnormality (step 102). The flag determination unit 18 determines whether or not there is an abnormality in the processing block 40 based on the flag added to the processing block 40.

[0108] For example, if it is determined that there is no abnormality in the processing block 40 (No in step 102), the whole image generating unit 19 generates image data 41b of the whole image 2 (step 103).

[0109] For example, if it is determined that there is an abnormality in the processing block 40 (Yes in step 102), the overall image generation unit 19 and the partial image generation unit 20 generate image data 41b of the overall image 2 and image data 41c of the partial image 3, respectively (step 104).

[0110] When the image data 41b of the entire image 2 and the image data 41c of the partial image 3 are generated, the data arranging unit 21 generates the insert data 4 (step 105).

[0111] 6 is a diagram showing the operation of each block of the insert data generation unit 14 from step 101 to step 105. The flow up to the generation of the insert data 4 will now be described with reference to FIG. 6. The numbers of the processing blocks 40 shown in FIG. 6 indicate the order in which the processing was performed.

[0112] If an abnormality is detected in the processing block 40 output from the abnormality detection unit 17, a flag is added. For example, no abnormality is detected in the first processing block 40. In this case, the flag determination unit 18 determines that there is no abnormality in the first processing block 40 (No in step 102), and the entire image generation unit 19 generates image data 41b of the entire image 2 without thinning out the first processing block 40 (step 103). This image data 41b is arranged in the image data area 46 by the data arrangement unit 21 (step 105).

[0113] Similarly, for the second processing block 40, image data 41b of the entire image 2 is generated without thinning and is arranged in the image data area 46. Note that for the first and second processing blocks 40, image data 41c of the partial image 3 is not generated.

[0114] 6, an abnormality is detected in the third processing block 40, and a flag is added. Therefore, the flag determination unit 18 determines that there is an abnormality in the third processing block 40 (Yes in step 102). In this case, the partial image generation unit 20 generates image data 41c of partial image 3 without thinning out the data for the third processing block 40 (step 104). Also, the entire image generation unit 19 thins out the data for the third processing block 40, and generates image data 41b of entire image 2 (step 104).

[0115] 7A and 7B are schematic diagrams showing an example of image thinning processing. Pixels 55 included in one row (line) of image data 41a of the captured image 1 are schematically shown as square blocks in Fig. 7A and Fig. 7B. Of these, hatched pixels 55 are pixels 55 used as image data, and white pixels 55 are pixels 55 that are thinned out by the thinning processing.

[0116] For example, in Fig. 7A, the thinning ratio is set to 1 / 2. In this case, every other pixel 55 in each row is selected, and the other pixels are thinned out. In Fig. 7B, the thinning ratio is set to 1 / 4. In this case, every fourth pixel 55 in each row is selected, and the other pixels are thinned out.

[0117] The method for thinning out pixels is not limited. For example, when the thinning ratio is 1 / 2, if even-numbered columns are thinned out on one line, odd-numbered columns may be thinned out on the next line, and so on. Similarly, when the thinning ratio is 1 / 4, the pixel columns to be thinned out may be changed on a row-by-row basis. This makes it possible to suppress the occurrence of vertical stripes, etc., that occur when pixels in each row are thinned out. Other than this, the method for thinning out pixels is not limited.

[0118] 6, when step 104 is completed for the third processing block 40 determined to have an abnormality, image data 41b (thinning ratio 1 / 2) of the entire image 2 and image data 41c of the partial image 3 are arranged in this order in the image data area 46. At this point, half the amount of extra data of the processing block 40 has been added to the image data area 46.

[0119] 6 , since no abnormality is detected in the fourth processing block 40 (No in step 102), the overall image generation unit 19 generates image data 41b of the overall image 2 for the fourth processing block 40 (step 103). In this case, to offset the above-mentioned excess data amount, the image data 41b of the overall image 2 for the fourth processing block 40 is thinned out at a rate of 1 / 2.

[0120] In this way, even if there is no abnormality in the processing block 40, the image data 41b of the entire image 2 is thinned out depending on the number of times the image data 41c of the partial image 3 is inserted (the number of times an abnormality is detected). The number of times an abnormality is detected is counted, for example, by the flag determination unit 18 as a consecutive abnormality counter.

[0121] For example, when the consecutive anomaly counter is 2 (when anomalies occur twice in a row), an amount of extra data equivalent to one processing block 40 is added, including the partial image 3 and the entire image 2. Therefore, in the processing blocks 40 in which no subsequent anomalies are detected, the image data 41b is thinned out by half twice in succession.

[0122] Furthermore, if the consecutive anomaly counter exceeds a certain number, it is possible that the amount of data will exceed the limit even if the image data 41b of the normal processing block 40 is thinned out. In this case, for the normal processing block 40, the image data 41b of the entire image 2 may not be generated in step 103, that is, the image data 41b of that processing block 40 may be deleted (see FIG. 3).

[0123] In this way, starting from the abnormal processing block 40, the number of whole images 2 that need to be thinned or deleted is counted for the subsequent processing blocks 40 according to the set thinning ratio. This makes it possible to maintain the amount of data 4 (image frames 6) to be inserted.

[0124] 7 , after step 101, metadata 5 for the processing block 40 is generated by the metadata generation unit 15, separately from the processes from step 102 to step 105. Specifically, from the information generated by the anomaly detection unit 17, the determination information generation unit 22, the type information generation unit 23, and the position information generation unit 24 generate anomaly determination information 47, anomaly type information 50, and position information 48 of the image data 41c of the partial image 3, respectively. These pieces of information are then stored sequentially in the metadata area 45.

[0125] Next, it is determined whether the processing block 40 is the last (step 107). For example, if an unprocessed processing block 40 remains (No in step 107), the process returns to step 101, and processing of the next processing block 40 begins. If the processing block 40 is the last block (Yes in step 107), the data output unit 16 outputs the image frame 6 including the insert data 4 and the metadata 5 (step 108).

[0126] Fig. 8 is a schematic diagram showing another example of an image frame. In Fig. 3 and Fig. 6, an example was described in which image data 41b of the entire image 2 is not generated when a series of abnormalities occur. Fig. 8 shows an example of settings to avoid such a situation.

[0127] Specifically, the thinning ratios of all image data included in the insertion data 4 (image data 41b of the entire image 2 for blocks other than the abnormal block 40e, image data 41b of the entire image 2 for the abnormal block 40e, and image data 41c of the partial image 3 for the abnormal block 40e) are set to be changeable. That is, in Fig. 8, the image data 41b of the entire image 2 and the image data 41c of the partial image 3 are generated by thinning out pixels of the captured image 1.

[0128] For example, if the amount of data in the processing block 40 is not exceeded by the image data 41c of the partial image 3 alone, the remaining amount of data is used for the image data 41b of the entire image 2. This makes it possible to generate the image data 41b of the entire image 2 even if abnormalities are detected consecutively.

[0129] For example, pixels of the captured image 1 are thinned out so that the total data amount of the image data 41b of the entire image 2 and the image data 41c of the partial image 3 in the abnormal block 40e fits within the data amount of the abnormal block 40e (the data amount of the processing block 40). As a result, as shown in Fig. 8, the image data 41b and image data 41c generated for each abnormal block 40e do not extend beyond the abnormal block 40e (thereby preventing the generation of excess data).

[0130] This method eliminates the need to offset the amount of data outside the abnormal block 40e, so for example, for processing blocks 40 other than the abnormal block 40e, the thinning ratio is set to 1, and image data 41b of the entire image 2 is generated without thinning.

[0131] 8, the thinning ratio for the entire image 2 and the partial image 3 of the abnormal block 40e is set to 1 / 2. That is, the pixels of the captured image 1 of the abnormal block 40e are thinned out by 1 / 2 to generate image data 41b of the entire image 2 and image data 41c of the partial image 3. This makes it possible to unify the resolution and speed up subsequent image processing, etc.

[0132] In addition, the thinning ratios for the entire image 2 and the partial image 3 are not limited. For example, the thinning ratio for the entire image 2 may be set to 1 / 4, and the thinning ratio for the partial image 3 may be set to 3 / 4. This makes it possible to improve the resolution of the partial image 3 while avoiding loss of the entire image 2. Furthermore, the thinning ratio for the entire image 2 for the processing blocks 40 other than the abnormal block 40e may be set to less than 1. This makes it less likely that the entire image 2 will be deleted even if the image data 41b and image data 41c extend beyond the abnormal block 40e.

[0133] Fig. 9 is a schematic diagram showing another example of an image frame. In the examples shown in Fig. 3 and Fig. 8, the image frame 6 was configured so that the amount of data was within a predetermined value and so as not to degrade the frame rate. In contrast, Fig. 9 shows an example of settings that prioritize the image quality (resolution) of the entire image 2 rather than the frame rate.

[0134] Specifically, in each row where image data 41b and image data 41c are arranged, an excess of data due to the insertion of image data 41c is permitted. In other words, if the amount of data in image data area 46 of image frame 6 exceeds a predetermined value due to the insertion of image data of partial image 3, the amount of data in the image data area is increased by the amount that exceeds the predetermined value.

[0135] 9, image data 41b (thinning ratio 1 / 2) of the entire image 2 and image data 41c of the partial image 3 without thinning are generated for both the processing block (5,3) and the processing block (6,1), which are abnormal blocks 40e. Furthermore, image data 41b (thinning ratio 1 / 2) of the entire image 2 of the processing block (6,2), which was deleted in FIG. 3, is inserted. As a result, the image data area 46 in the sixth row is expanded by half the amount of data of the processing block 40.

[0136] In this way, increasing the amount of data in the image data area 46 may result in a decrease in the frame rate. On the other hand, since it is not necessary to delete the image data 41b of the entire image 2, there is no loss of the entire image 2, and it is possible to improve the image quality. In addition, it may be possible to switch appropriately by mode selection or the like between a setting that prioritizes image quality as shown in Fig. 9 and a setting that prioritizes frame rate as shown in Fig. 3 or 8.

[0137] 10 is a flowchart showing an example of the operation of the display controller. The following describes basic processing performed by the display controller 33 when it receives the image frame 6. This processing is a loop process performed for each row (line) of the image frame 6.

[0138] First, the image separator 35 acquires row data of the image frame 6 via the data receiver 34 (step 201). The row data is data for each row of the image frame 6, and includes metadata 5 and line-shaped image data (pixel columns) (see FIG. 4).

[0139] Next, the image separation unit 35 determines whether each piece of line-shaped image data is data for a partial image (step 202). In this process, the image separation unit 35 references the insertion position of the partial image 3 (the start pixel position and the end pixel position of each piece of image data 41c for the partial image 3) recorded in the metadata 5, and separates the image data 41b for the entire image 2 and the image data 41c for the partial image 3 from the line-shaped image data.

[0140] For example, if it is determined that the image data is not data for partial image 3 (No in step 202), the image data is considered to be image data 41b for entire image 2 and is written to a memory area for generating entire image 2 (step 203).

[0141] For example, if it is determined that the image data is data for partial image 3 (Yes in step 202), the image data is considered to be image data 41c for partial image 3 and is written to a memory area for generating partial image 3 (step 204).

[0142] When step 203 or step 204 is completed, it is determined whether the data for image frame 6 has been completed (step 205). Specifically, it is determined whether the processed row data is the last row data for image frame 6. For example, if unprocessed row data remains (No in step 205), the next row data for which the processing from step 101 onwards is executed is processed.

[0143] If the row data is the last row data (Yes in step 205), both the process of generating whole image 2 in step 203 and the process of generating partial image 3 in step 204 are completed. In this case, the display image generating unit 36 ​​generates a display image including whole image 2 and partial image 3 (step 206).

[0144] 11 is a schematic diagram showing an example of a display image 60 displayed on the display unit. The display image 60 is an image that displays an entire image 2 and a partial image 3 of an object, and is configured as, for example, a GUI (such as an inspection screen) that allows a user to check the state of the object.

[0145] 11 , the entire image 2 is displayed on the left side of the display image 60, and the partial image 3 is displayed on the right side. The range displayed as the partial image 3 is also displayed as a dotted line area in the entire image 2. In this way, the entire image 2 and the partial image 3 are displayed simultaneously on the same screen, so that the user can easily check both the overall condition of the object and the condition of the abnormal region 38.

[0146] It is also possible to display a separate partial image 3 for each abnormal region 38 that has occurred in the object. For example, a separate partial image 3 may be displayed for each of the upper and lower abnormal regions 38 in the drawing. The process of separating the partial images 3 is performed, for example, by analyzing the shape of the partial images 3 in the display image generation unit 36. Alternatively, the imaging controller 13 may separate and generate the partial images 3 for each abnormal region 38.

[0147] In this way, the display controller 33 separates the entire image 2 and the partial image 3 in parallel from one image frame 6 (insertion data 4). This makes it possible to synchronize the timing of image processing and the like for the entire image 2 and the partial image 3. Furthermore, it is no longer necessary to provide a buffer for storing one image while waiting for the other to be transmitted, for example. This makes it possible to process multiple images with a simple configuration.

[0148] As described above, the imaging controller 13 according to this embodiment outputs an image frame 6 including insertion data 4 in which image data 41c of at least one partial image 3 is inserted between image data 41b of the entire image 2, and metadata 5 including the insertion position of the image data 41c of the partial image 3. This makes it possible to transmit data for multiple images in one image frame 6. Furthermore, the display controller 33 according to this embodiment can separate each image in parallel from the image frame 6 by referring to the metadata 5. As a result, it is possible to simplify the configuration of the inspection system 100, which is a system for transmitting multiple images.

[0149] Fig. 12 is a schematic diagram showing an example of the configuration of an inspection system as a comparative example. The inspection system 101 shown in Fig. 12 is provided with two output interfaces 25 (output IF1 and output IF2), and an entire image 2 and a partial image 3 are transmitted using each output interface 25, respectively.

[0150] For example, data of the entire image 2 and the partial image 3 are generated by the image generation unit 114 of the imaging controller 113 and supplied to the data output unit 116. The data output unit 116 generates an image frame of the entire image 2 and an image frame of the partial image 3, and transmits them to the display controller 33 using the output IF1 and output IF2.

[0151] The display controller 133 is provided with a buffer 120 for the entire image 2 and a buffer 120 for the partial image 3 at the downstream of the data receiving unit 134. The storage capacity of each buffer 120 is set to a size that can store, for example, one or more images (the entire image 2 and the partial image 3). This allows the display image generating unit 136 to simultaneously process the entire image 2 and the partial image 3 even if the transmission timings of these images are different.

[0152] In this way, when multiple images are transmitted through different output interfaces 25, a downstream circuit or processor (display controller 133 in this case) provides a buffer 120 for each output interface 25, and performs processing to display the images after processing to synchronize the timing of each data. This method requires providing multiple output interfaces 25 and downstream buffers 120, which is likely to increase the cost of implementing the system. Furthermore, the output interface 25 (output IF2) that transmits partial image 3 is not used unless an abnormality is detected, which means that the total output performance is wasted.

[0153] Another possible method for transmitting the entire image 2 and the partial image 3 is to transmit the image frame of the entire image 2 first, followed by the image frame of the partial image 3. In this case, it is possible to transmit each image using a single output interface 25, but a buffer of sufficient size must be provided at the subsequent stage to display the entire image 2 and the partial image 3 together.

[0154] In this embodiment, insertion data 4, in which image data 41c of partial images 3 are inserted between image data 41b of entire images 2, is transmitted as one image frame together with metadata 5 indicating the insertion position. This makes it possible to transmit, for example, two types of images using one output interface 25. Furthermore, since there is no need to increase the number of output interfaces 25, the total output performance is not wasted and system costs can be reduced.

[0155] Furthermore, when performing display processing or the like at a later stage, it is desirable to have a small time lag between the corresponding data of the entire image 2 and the partial image 3. In this regard, in this embodiment, by configuring the image frame 6 as described above, it is possible to transmit the corresponding data of the entire image 2 and the partial image 3 almost continuously. Therefore, the time lag required for transmitting each image is minimized.

[0156] Furthermore, since the entire image 2 and the partial image 3 can be restored in parallel almost simultaneously, there is no need to prepare a large amount of buffers in the subsequent stages. For example, depending on how the inspection system 100 is used, it may be possible to eliminate processing using a buffer in the subsequent stages. In this case, it is possible to improve the frame rate.

[0157] In this embodiment, data obtained by thinning out the image data 41a of the original captured image 1 is used as the image data 41b of the entire image 2 and the image data 41c of the partial image 3. This makes it possible to keep the data volume of, for example, the image frame 6 within a predetermined value. As a result, it is possible to simplify the system, including the downstream circuitry, while maintaining the same frame rate as when transmitting the captured image 1.

[0158] The thinning ratio can be changed as needed to suit the needs of the user, which allows each user to select the optimal balance of resolution between the entire image 2 and the partial image 3, thereby improving the usability of the inspection system 100.

[0159] Second Embodiment An image display device according to a second embodiment of the present technology will be described. In the following description, descriptions of parts having the same configurations and functions as those of the inspection system 100 described in the above embodiment will be omitted or simplified.

[0160] In the above embodiment, a configuration has been described in which the entire image 2 and the partial image 3 generated from the captured image 1 are transmitted as one image frame 6. The present technology is not limited to the entire image 2 and the partial image 3, and can be applied to any system that transmits multiple images.

[0161] 13 is a schematic diagram showing an example of the configuration of an image transmission system according to the second embodiment. The image transmission system 200 has a transmission controller 213 and a display controller 233, and is a system that transmits two types of image data from the transmission controller 213 to the display controller 233 via a single output interface 25. In this embodiment, the transmission controller 213 corresponds to an information processing device that outputs image frames, and the display controller 233 corresponds to an information processing device that receives the image frames.

[0162] The transmission controller 213 has, as functional blocks, an insert data generation unit 214, a metadata generation unit 215, and a data output unit 216.

[0163] The insertion data generation unit 214 reads the first input image 7 a and the second input image 7 b. The types and generation methods of the first input image 7 a and the second input image 7 b are not limited. For example, each image may be generated by an image processing block (not shown), or each image may be read from a memory element such as a buffer.

[0164] The insertion data generation unit 214 also generates a first image 8a by thinning out pixels from the first input image 7a, and a second image 8b by thinning out pixels from the second input image 7b. The insertion data generation unit 214 also generates insertion data 4 by inserting image data of a second image 8b, which is different from the first image 8a, between the image data of the first image 8a. In this way, in this embodiment, the first image 8a and the second image 8b, which are thinned images generated by thinning out the first input image 7a and the second input image 7b, are the images transmitted by the image frame 6 (insertion data 4).

[0165] As shown in FIG. 13, the insert data generation unit 214 includes a first image generation unit 219 , a second image generation unit 220 , and a data arrangement unit 221 .

[0166] The first image generation unit 219 thins out the image data of the first input image 7a in processing block units according to the first thinning ratio to generate image data of a first image 8a. The second image generation unit 220 thins out the image data of the second input image 7a in processing block units according to the second thinning ratio to generate image data of a second image 8b. The first thinning ratio, the second thinning ratio, and the size of the processing block are stored as setting parameters in a storage unit (not shown), for example.

[0167] The data arrangement unit 221 arranges the image data of the first image 8a generated by the first image generation unit 219 and the image data of the second image 8b generated by the second image generation unit 220 in accordance with the sorting start coordinates stored as setting parameters to generate the insertion data 4. The sorting start coordinates include, for example, the insertion positions of each image data of the second image 8b generated in processing block units. The image data of the first image 8a and the second image 8b are arranged in order within the image data area set in the image frame 6 so that the image data of the second image 8b is inserted at these insertion positions.

[0168] The metadata generation unit 215 generates metadata 5 corresponding to the insert data 4. Specifically, the metadata generation unit 215 generates metadata 5 including the insertion position of the image data of the second image 8b in the insert data 4. As shown in FIG. 13 , the metadata generation unit 215 includes a position information generation unit 224.

[0169] The position information generator 224 reads the size of the processing block and the rearrangement start coordinates recorded as setting parameters. Based on this information, the position information generator 224 generates position information indicating the insertion position of the image data of the second image 8b in the insertion data 4. The position information records, for example, pixel positions indicating the range of the image data of the second image 8b.

[0170] The data output unit 216 outputs an image frame 6 including the insert data 4 and the metadata 5. Specifically, the data output unit 216 integrates the insert data 4 output from the data arrangement unit 221 and the metadata 5 including the information (position information) output from the metadata generation unit 215 into one image frame 6. The generated image frame 6 is transmitted to the display controller 233 via one output interface 25 (output IF).

[0171] The display controller 233 has, as functional blocks, a data receiving unit 234 , an image separating unit 235 , and a display image generating unit 236 .

[0172] The data receiving unit 234 receives an image frame 6 including insertion data 4 in which image data of a second image 8b other than the first image 8a is inserted between image data of the first image 8a, and metadata 5 including the insertion position of the image data of the second image 8a in the insertion data 4.

[0173] The image separation unit 235 separates the image data of the first image 8 a and the second image 8 b from the insertion data 4 based on the metadata 5 .

[0174] The display image generation unit 236 generates the first image 8 a and the second image 8 b using the image data of the first image 8 a and the second image 8 b. The display image generation unit 236 also generates a display image such as a GUI including the first image 8 a and the second image 8 b, and outputs the image to a display device (not shown).

[0175] In this way, in the image transmission system 200, the image frame 6 including the first image 8a and the second image 8b is transmitted using one output interface 25, so there is no need to prepare a separate output interface 25 for transmitting each image. Furthermore, when the image frame 6 is received, the image data of the first image 8a and the second image 8b are output alternately, so the time lag required for transmitting each image is minimized and there is no need to prepare a large amount of buffers.

[0176] The image transmission system 200 can be applied to, for example, a dual-stream system that simultaneously displays an entire image of an observation area (first image 8a) and an ROI (Range of Interest) image (second image 8b). Use of the image transmission system 200 can simplify the configuration of such a system, thereby reducing the cost required to realize the system.

[0177] Other Embodiments The present technology is not limited to the above-described embodiments, and various other embodiments can be realized.

[0178] 13 , the original images (first input image 7a and second input image 7b) are thinned out to generate two images (first image 8a and second image 8b) to be stored in image frame 6. For example, by expanding the amount of data in image frame 6, it is possible to generate image frame 6 without thinning out the original image. This reduces the frame rate, but makes it possible to transmit the first input image 7a and the second input image 7b through a single output interface without degrading the image quality.

[0179] Furthermore, in the above embodiment, an image frame has been described in which image data of one image (the entire image 2 or the first image 8a) is inserted between the image data of another image (the partial image 3 or the second image 8b). In this case, the insertion position of the image data of the other image is recorded as metadata, making it possible to separate the image data of the two images. For example, it is also possible to insert image data of two or more other images between the image data of one image. In this case, the information about the insertion position in the metadata stores an index or the like for distinguishing the inserted images. By referencing this index, it is possible to properly separate the image data of each image, even in an image frame containing image data of three or more images.

[0180] It is also possible to combine at least two of the features of the present technology described above. That is, the various features described in each embodiment may be arbitrarily combined without distinguishing between the embodiments. Furthermore, the various effects described above are merely examples and are not intended to be limiting, and other effects may also be achieved.

[0181] In the present disclosure, the terms "same," "equal," "orthogonal," etc. are concepts that include "substantially the same," "substantially equal," "substantially orthogonal," etc. For example, they also include states that fall within a predetermined range (e.g., a range of ±10%) based on "completely the same," "completely equal," "completely orthogonal," etc.

[0182] Note that the present technology can also be configured as follows. (1) An information processing device comprising: a first generating unit that generates insertion data in which image data of at least one second image different from the first image is inserted between image data of the first image; a second generating unit that generates metadata including an insertion position of the image data of the second image in the insertion data; and an output unit that outputs an image frame including the insertion data and the metadata. (2) The information processing device described in (1), in which the image data of at least one of the first image and the second image includes image data in which pixels of an original image have been thinned out. (3) The information processing device described in (1) or (2), in which the image data of the first image and the image data of the second image are each configured as pixel blocks, and the insertion data is data in which the pixel blocks are arranged in an image data area of ​​the image frame. (4) The information processing device described in at least one of (1) to (3), in which the insertion data is data in a matrix format, and the second generating unit generates the metadata for each row of the insertion data. (5) The information processing device according to at least one of (1) to (4), wherein the first image is a whole image representing the whole of an input image, the second image is a partial image representing a part of the input image, and the first generation unit generates the insertion data by inserting image data of the partial image between image data of the whole image at a position corresponding to the partial image. (6) The information processing device according to (5), wherein the first generation unit determines the presence or absence of an abnormality in the input image in units of processing blocks of a predetermined size, and generates image data of the partial image from image data of the input image in an abnormal block that is the processing block determined to have the abnormality. (7) The information processing device according to (6), wherein the first generation unit generates image data of the partial image using all pixels of the input image in the abnormal block.(8) The information processing device according to (6) or (7), wherein the first generation unit generates image data of the entire image by thinning out pixels of the input image in the abnormal block. (9) The information processing device according to (8), wherein the first generation unit deletes a part of the image data of the entire image when inserting the image data of the partial image causes the amount of data in the image data area of ​​the image frame to exceed a predetermined value. (10) The information processing device according to (8), wherein the first generation unit increases the amount of data in the image data area by an amount that exceeds the predetermined value when inserting the image data of the partial image causes the amount of data in the image data area of ​​the image frame to exceed a predetermined value. (11) The information processing device according to (6), wherein the first generation unit generates the image data of the entire image and the image data of the partial image by thinning out pixels of the input image. (12) The information processing device according to (11), wherein the first generation unit thins out pixels of the input image so that the total data amount of the image data of the entire image and the partial image in the abnormal block falls within the data amount of the abnormal block. (13) The information processing device according to (12), wherein the first generation unit thins out pixels of the input image in the abnormal block by half to generate image data of the entire image and the partial image. (14) The information processing device according to at least one of (6) to (13), wherein the metadata includes a determination result of the presence or absence of the abnormality. (15) The information processing device according to at least one of (6) to (14), wherein the metadata includes a type of the abnormality. (16) The information processing device according to at least one of (1) to (15), wherein the input image is an image captured by a solid-state imaging element, and the information processing device is configured as a sensor unit including the solid-state imaging element.(17) An information processing device comprising: a receiving unit that receives an image frame including insertion data in which image data of at least one second image other than the first image is inserted between image data of the first image, and metadata including an insertion position of the image data of the second image in the insertion data, and a separating unit that separates each of the image data of the first image and the second image from the insertion data based on the metadata. (18) A data structure used in transmission processing of the first image and the second image, including insertion data in which image data of at least one second image other than the first image is inserted between image data of the first image, and metadata including an insertion position of the image data of the second image in the insertion data.

[0183] DESCRIPTION OF SYMBOLS 1...Captured image 2...Full image 3...Partial image 4...Insertion data 5...Metadata 6...Image frame 10...Image sensor unit 11...Image sensor 13...Image capture controller 14...Insertion data generation section 15...Metadata generation section 16...Data output section 25...Output interface 33...Display controller 34...Data receiving section 35...Image separation section 100...Inspection system

Claims

1. An information processing device comprising: a first generation unit that generates insertion data by inserting image data of at least one second image other than a first image between image data of a first image; a second generation unit that generates metadata including an insertion position of the image data of the second image in the insertion data; and an output unit that outputs an image frame including the insertion data and the metadata.

2. An information processing device according to claim 1, wherein the image data of at least one of the first image and the second image includes image data in which pixels of an original image have been thinned out.

3. An information processing device according to claim 1, wherein the image data of the first image and the image data of the second image are each configured as pixel blocks, and the insertion data is data in which the pixel blocks are arranged in an image data area of ​​the image frame.

4. An information processing device according to claim 1, wherein the insert data is data in a matrix format, and the second generating unit generates the metadata for each row of the insert data.

5. An information processing device as described in claim 1, wherein the first image is an overall image representing the entire input image, the second image is a partial image representing a part of the input image, and the first generation unit generates the insertion data by inserting image data of the partial image between image data of the overall image at a position corresponding to the partial image.

6. An information processing device according to claim 5, wherein the first generation unit determines the presence or absence of an abnormality in the input image in units of processing blocks of a predetermined size, and generates image data of the partial image from image data of the input image in an abnormal block, which is the processing block determined to have the abnormality.

7. An information processing device according to claim 6, wherein the first generating unit generates image data of the partial image by using all pixels of the input image in the abnormal block.

8. An information processing device according to claim 6, wherein the first generating section generates image data of the entire image by thinning out pixels of the input image in the abnormal block.

9. An information processing device according to claim 8, wherein the first generation unit deletes a portion of the image data of the entire image when inserting image data of the partial image causes the amount of data in the image data area of ​​the image frame to exceed a predetermined value.

10. An information processing device as described in claim 8, wherein the first generation unit increases the amount of data in the image data area of ​​the image frame by an amount that exceeds a predetermined value when inserting image data of the partial image causes the amount of data in the image data area of ​​the image frame to exceed a predetermined value.

11. An information processing device according to claim 6, wherein the first generating unit generates the image data of the entire image and the image data of the partial image by thinning out pixels of the input image.

12. An information processing device according to claim 11, wherein the first generation unit thins out pixels of the input image so that the total data amount of the image data of the entire image and the image data of the partial image in the abnormal block falls within the data amount of the abnormal block.

13. An information processing device according to claim 12, wherein the first generating unit generates image data of the entire image and image data of the partial image by thinning out the pixels of the input image in the abnormal block by half.

14. An information processing device according to claim 6, wherein the metadata includes a result of determining whether or not the abnormality exists.

15. An information processing device according to claim 6, wherein the metadata includes a type of the anomaly.

16. An information processing device according to claim 1, wherein the input image is an image captured by a solid-state imaging element, and the information processing device is configured as a sensor unit including the solid-state imaging element.

17. An information processing device comprising: a receiving unit that receives an image frame including insertion data in which image data of at least one second image other than a first image is inserted between image data of a first image, and metadata including an insertion position of the image data of the second image in the insertion data; and a separation unit that separates each of the image data of the first image and the second image from the insertion data based on the metadata.

18. A data structure used in transmission processing of the first image and the second image, comprising insertion data in which image data of at least one second image other than the first image is inserted between image data of the first image, and metadata including the insertion position of the image data of the second image in the insertion data.

Images