Receiving device and transmission system

The described system efficiently processes and displays ROI images by using a receiving device and transmission system with embedded data and dummy image insertion, addressing the inefficiencies in existing technologies and enhancing transmission efficiency.

JP7584497B2Active Publication Date: 2024-11-15SONY SEMICON SOLUTIONS CORP
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
JP2022509417
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-03-25
Filing Date
2021-02-19
Publication Date
2024-11-15
Estimated Expiration
2041-02-19

AI Technical Summary

Technical Problem

Existing technologies do not efficiently process and display a region of interest (ROI) cut out from a captured image on a display device, particularly in transmission systems handling large data loads.

Method used

A receiving device and transmission system that includes a receiving unit, frame memory, and processing unit to restore ROI image data based on embedded information, with dummy image data insertion to manage memory efficiently, and a transmitting device that transmits ROI image data and information as payload and embedded data.

Benefits of technology

Enables efficient display of ROI images even under various constraints, achieving high transmission efficiency by processing speed increases and reducing the need for low power modes during multiple ROI transmissions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007584497000001
    Figure 0007584497000001
  • Figure 0007584497000002
    Figure 0007584497000002
  • Figure 0007584497000003
    Figure 0007584497000003
Patent Text Reader

Abstract

The objective of the present invention is to make it possible to efficiently display, on a display device, image data for some regions of interest (ROI) cut out from a captured image. A reception device comprises: a reception unit that receives a transmittal signal including, in payload data, image data for a prescribed number of regions of interest (ROI) cut out from a prescribed captured image, and including, in embedded data, ROI information corresponding to the prescribed number of ROIs; and a processing unit that can execute a restoration process for restoring the ROI image data, on the basis of the ROI information extracted from the embedded data, to a state in which the ROI image data can be displayed on a prescribed display device.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a receiving device and a transmission system. [Background technology]

[0002] In recent years, there has been an increase in applications that require the transmission of large amounts of data. This puts a heavy load on the transmission system, and in the worst case scenario, the transmission system may crash, making it impossible to transmit data.

[0003] In order to avoid a breakdown in the transmission system, it is known that, for example, instead of transmitting all captured images, the object being photographed is identified and only a portion of the image cut out from the identified object is transmitted (e.g., Patent Documents 1 to 4). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-201756 A [Patent Document 2] JP 2014-39219 A [Patent Document 3] JP 2013-164834 A [Patent Document 4] JP 2012-209831 A Summary of the Invention [Problem to be solved by the invention]

[0005] No consideration has been given to a process for efficiently displaying a part of a region of interest (ROI (Region of Interest)) that has been cut out from a captured image and transmitted on a display device. An object of the present disclosure is to provide a receiving device and a transmission system that can efficiently display a portion of a region of interest (ROI) cut out from a captured image on a display device. [Means for solving the problem]

[0006] A receiving device according to an embodiment of the present disclosure includes a receiving unit that receives a transmission signal including image data of a predetermined number of ROIs (Regions of Interest) cut out from a predetermined captured image in payload data and ROI information corresponding to the predetermined number of ROIs in embedded data; A frame memory capable of storing image data of the ROI; A processing unit capable of executing a restoration process for restoring the image data of the ROI to a state that can be displayed on a predetermined display device based on the ROI information extracted from the embedded data. The processing unit writes image data of the ROI in units of payload data to the frame memory based on ROI restoration information generated from the ROI information, and when a predetermined condition is satisfied, generates dummy inserted image data by inserting dummy image data between image data of two or more ROIs in the payload data and connecting the image data of the two or more ROIs, and writes the generated dummy inserted image data continuously to the frame memory in response to a single instruction, and the predetermined condition is that a gap between the image data of the two or more ROIs is narrow and a memory size of the dummy image data to be inserted into the gap is within a predetermined size. do.

[0007] A transmission system according to one aspect of the present disclosure includes a transmitting device that transmits image data of a region of interest (ROI) as payload data and transmits ROI information as embedded data, and a receiving unit that receives a transmission signal that includes image data of a predetermined number of ROIs cut out from a predetermined captured image in the payload data and includes ROI information corresponding to the predetermined number of ROIs in the embedded data; a frame memory capable of storing image data of the ROI; and a receiving device having a processing unit capable of executing a restoration process for restoring the image data of the predetermined number of ROIs to a state that can be displayed on a predetermined display device based on the ROI information extracted from the embedded data. The receiving device, in the processing unit, writes image data of the ROI to the frame memory in units of the payload data based on ROI restoration information generated from the ROI information, and when a predetermined condition is satisfied, generates dummy-inserted image data by inserting dummy image data between image data of two or more ROIs in the payload data and connecting the image data of the two or more ROIs, and writes the generated dummy-inserted image data continuously to the frame memory in response to a single instruction, and the predetermined condition is that a gap between the image data of the two or more ROIs is narrow and a memory size of the dummy image data to be inserted into the gap is within a predetermined size. do. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a video transmission system. [Diagram 2] 2 is a diagram illustrating an example of a schematic configuration of the video transmission device of FIG. 1. [Diagram 3] 11 is a diagram illustrating an example of a procedure for generating transmission data when a captured image includes two ROIs. FIG. [Figure 4] FIG. 13 is a diagram illustrating an example of the configuration of a packet header. [Diagram 5] FIG. 2 is a diagram illustrating an example of a configuration of transmission data. [Figure 6]FIG. 2 is a diagram illustrating an example of a configuration of transmission data. [Figure 7] FIG. 13 is a diagram illustrating an example of the configuration of payload data of a long packet. [Figure 8] 2 is a diagram illustrating an example of a schematic configuration of the video receiving device of FIG. 1. [Figure 9] 11 is a diagram showing an example of a procedure for generating two ROI images included in a captured image when two images are included in transmission data. FIG. [Figure 10] FIG. 10 is a diagram illustrating a schematic area in which an object identified in a captured image is located. [Figure 11] FIG. 13 is a diagram showing an example of a ROI region set for a specified object. [Figure 12] 13 is a diagram showing an example of the configuration of transmission data in which position information of an ROI image is included in payload data of a long packet. FIG. [Figure 13] FIG. 2 is a diagram illustrating an example of a captured image according to an embodiment of the present disclosure. [Figure 14] FIG. 14 is a diagram showing the coordinates and image size of the region of interest shown in FIG. 13. [Figure 15] 14 is a diagram showing an example of payload data including image data of the region of interest shown in FIG. 13. [Figure 16] 4A to 4C are diagrams illustrating first coordinate data and second coordinate data constituting ROI restoration information according to an embodiment of the present disclosure. [Figure 17] 11A and 11B are diagrams illustrating an example of X-coordinate ordered data and Y-coordinate ordered data according to the embodiment of the present disclosure. [Figure 18] 1 is a block diagram showing a schematic configuration of a video transmission system according to an embodiment of the present disclosure. [Figure 19] 1 is a diagram illustrating an example of a schematic configuration of a video receiving device according to an embodiment of the present disclosure. [Figure 20] 11 is a flowchart showing an example of a flow of a restoration process in a video receiving device according to the present embodiment of the present disclosure. [Figure 21] 13 is a flowchart showing an example of the flow of image data replacement processing in the video receiving device 200a according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The following description is a specific example of the present disclosure, and the present disclosure is not limited to the following embodiment.

[0010] Hereinafter, modes for carrying out the technology according to the present disclosure (hereinafter, referred to as "embodiments") will be described in the following order. 1. Premise technology 1 of this disclosure (technology for transmitting a region of interest (ROI) that is a part (rectangular in shape) cut out from a captured image) 2. Premise Technology 2 of the Present Disclosure (Technology for transmitting a portion (non-rectangular shape) of a region of interest (ROI) cut out from a captured image) 3. Principle of restoration process in the embodiment of the present disclosure 4. Video receiving device and video transmission system according to embodiments of the present disclosure 5. Processing related to display of image data upon video reception according to an embodiment of the present disclosure 6. Modifications of the restoration process in the embodiment of the present disclosure

[0011] 1. Prerequisite technology 1 of this disclosure: [composition] In recent years, mobile devices such as smartphones and camera devices handle increasing amounts of image data, and there is a demand for faster data transmission within devices or between different devices with lower power consumption. To meet these demands, the standardization of high-speed interface standards such as the C-PHY standard and the D-PHY standard, established by the MIPI Alliance, is underway as connection interfaces for mobile devices and camera devices. The C-PHY standard and the D-PHY standard are interface standards for the physical layer (PHY) of communication protocols. In addition, there are DSI for displays of mobile devices and CSI for camera devices as upper protocol layers of the C-PHY standard and the D-PHY standard.

[0012] A video transmission system 1 according to the base technology of the present disclosure is a system capable of transmitting and receiving signals according to various standards, for example, the MIPICSI-2 standard, the MIPICSI-3 standard, or the MIPI DSI standard. FIG. 1 shows an overview of the video transmission system 1 according to the base technology of the present disclosure. The video transmission system 1 is applied to the transmission of a data signal, a clock signal, and a control signal, and includes a video transmitting device 100 and a video receiving device 200. The video transmission system 1 includes a data lane DL for transmitting a data signal such as image data, a clock lane CL for transmitting a clock signal, and a camera control interface CCI for transmitting a control signal, across the video transmitting device 100 and the video receiving device 200. FIG. 1 shows an example in which one data lane DL is provided, but multiple data lanes DL may be provided. The camera control interface CCI is a bidirectional control interface compatible with the I2C (Inter-Integrated Circuit) standard.

[0013] The video transmitting device 100 is a device that transmits signals in accordance with the MIP CSI-2 standard, the MIP CSI-3 standard, or the MIP DSI standard. It has a CSI transmitter 100A and a CCI slave 100B. The video receiving device 200 has a CSI receiver 200A and a CCI master 200B. In the clock lane CL, the CSI transmitter 100A and the CSI receiver 200A are connected by a clock signal line. In the data lane DL, the CSI transmitter 100A and the CSI receiver 200A are connected by a clock signal line. In the camera control interface CCI, the CCI slave 100B and the CCI master 200B are connected by a control signal line.

[0014] The CSI transmitter 100A is, for example, a differential signal transmission circuit that generates a differential clock signal as a clock signal and outputs it to a clock signal line. The CSI transmitter 100A can be configured to transmit not only differential signals but also single-ended or three-phase signals. The CSI transmitter 100A is also a differential signal transmission circuit that generates a differential data signal as a data signal and outputs it to a data signal line. The CSI receiver 200A is a differential signal receiving circuit that receives a differential clock signal as a clock signal via a clock signal line and performs a predetermined process on the received differential clock signal. The CSI receiver 200A is also a differential signal receiving circuit that receives a differential data signal as a data signal via a data signal line and performs a predetermined process on the received differential data signal.

[0015] (Video transmitting device 100) Fig. 2 shows an example of the configuration of video transmitting device 100. Video transmitting device 100 corresponds to a specific example of CSI transmitter 100A. Video transmitting device 100 includes, for example, an imaging unit 110, image processing units 120 and 130, and a transmission unit 140. Video transmitting device 100 transmits transmission data 147A generated by performing a predetermined process on captured image 111 obtained by imaging unit 110 to video receiving device 200 via a data lane DL. Fig. 3 shows an example of a procedure for generating transmission data 147A.

[0016] The imaging unit 110 converts an optical image signal obtained through, for example, an optical lens into image data. The imaging unit 110 includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging unit 110 has an analog-digital conversion circuit and converts analog image data into digital image data. The data format after conversion may be a YCbCr format in which the color of each pixel is expressed by a luminance component Y and color difference components Cb and Cr, or may be an RGB format. The imaging unit 110 outputs a captured image 111 (digital image data) obtained by imaging to the image processing unit 120.

[0017] The image processing unit 120 is a circuit that performs a predetermined process on the captured image 111 input from the imaging unit 110. In the base technology 1, a case will be described in which the image processing unit 120 performs a predetermined process on the captured image 111 input from the imaging unit 110 when a control signal instructing the cutting out of the ROI is input from the video receiving device 200 via the camera control interface CCI. However, the base technology 1 can also be applied to a case in which the video transmitting device 100, that is, the transmitting side, instructs the coordinates of the cutout of the ROI. In this case, the transmitting side is configured to receive information such as a "person" or "object" to be acquired in the ROI sent from the receiving side, and determine and instruct the coordinates of the cutout. As a result, the image processing unit 120 generates various data (120A, 120B, 120C) and outputs them to the transmitting unit 140. The image processing unit 130 is a circuit that performs a predetermined process on the captured image 111 input from the imaging unit 110. When a control signal instructing output of a normal image is input from the video receiving device 200 via the camera control interface CCI, the image processing unit 130 performs a predetermined process on the captured image 111 input from the imaging unit 110. As a result, the image processing unit 130 generates image data 130A and outputs it to the transmission unit 140.

[0018] The image processing unit 130 includes, for example, an encoding unit 131. The encoding unit 131 encodes the captured image 111 to generate compressed image data 130A. The image processing unit 130 compresses the captured image 111 in a compression format conforming to the JPEG (Joint Photographic Experts Group) standard as the format of the compressed image data 130A, for example.

[0019] The image processing unit 120 includes, for example, an ROI extraction unit 121, an ROI analysis unit 122, an overlap detection unit 123, a priority setting unit 124, an encoding unit 125, and an image processing control unit 126.

[0020] The ROI cutout unit 121 identifies one or more objects of the subject to be photographed included in the captured image 111 input from the imaging unit 110, and sets a region of interest ROI for each identified object. The region of interest ROI is, for example, a rectangular region including the identified object. The ROI cutout unit 121 cuts out an image of each region of interest ROI (for example, the ROI image 112 in FIG. 3) from the captured image 111. The ROI cutout unit 121 further assigns a region number as an identifier to each set region of interest ROI. For example, when two regions of interest ROI are set in the captured image 111, the ROI cutout unit 121 assigns region number 1 to one region of interest ROI (for example, the region of interest ROI1 in FIG. 3) and assigns region number 2 to the other region of interest ROI (for example, the region of interest ROI2 in FIG. 3). The ROI cutout unit 121 stores the assigned identifier (region number) in a storage unit, for example. The ROI cutout unit 121 stores, for example, each ROI image 112 cut out from the captured image 111 in the storage unit. The ROI cutout unit 121 further stores, for example, an identifier (region number) given to each region of interest ROI in the storage unit in association with the ROI image 112.

[0021] The ROI analysis unit 122 derives position information 113 of the region of interest ROI in the captured image 111 for each region of interest ROI. The position information 113 is composed of, for example, the upper left end coordinates (Xa, Ya) of the region of interest ROI, the length of the region of interest ROI in the X-axis direction, and the length of the region of interest ROI in the Y-axis direction. The length of the region of interest ROI in the X-axis direction is, for example, the physical region length XLa of the region of interest ROI in the X-axis direction. The length of the region of interest ROI in the Y-axis direction is, for example, the physical region length YLa of the region of interest ROI in the Y-axis direction. The physical region length refers to the physical length (data length) of the region of interest ROI. The position information 113 may include coordinates of a position different from the upper left end of the region of interest ROI. The ROI analysis unit 122 stores the derived position information 113 in, for example, a storage unit. The ROI analysis unit 122 stores the derived position information 113 in, for example, a storage unit in association with an identifier (region number) assigned to the region of interest ROI.

[0022] The ROI analysis unit 122 may further derive, for example, an output region length XLc in the X-axis direction of the region of interest ROI and an output region length YLc in the Y-axis direction of the region of interest ROI as the position information 113 for each region of interest ROI. The output region length is, for example, the physical length (data length) of the region of interest ROI after a resolution change is performed on the region of interest ROI by thinning processing, pixel addition, etc. The ROI analysis unit 122 may derive, for example, sensing information, exposure information, gain information, AD (Analog-Digital) word length, image format, etc., in addition to the position information 113 for each region of interest ROI, and store them in the storage unit.

[0023] The sensing information refers to the calculation contents for an object included in the region of interest ROI, supplementary information for subsequent signal processing of the ROI image 112, etc. The exposure information refers to the exposure time of the region of interest ROI. The gain information refers to the gain information of the region of interest ROI. The AD word length refers to the word length of data per pixel that has been AD converted in the region of interest ROI. The image format refers to the format of the image of the region of interest ROI. The ROI analysis unit 122 may, for example, derive the number of regions of interest ROI (number of ROIs) included in the captured image 111 and store it in the memory unit.

[0024] When a plurality of objects of the photographing target are specified in the captured image 111, the overlap detection unit 123 detects an overlap region (ROO (Region Of Overlap)) where two or more regions of interest ROI overlap each other based on the position information 113 of the plurality of regions of interest ROI in the captured image 111. That is, the overlap detection unit 123 derives the position information 114 of the overlap region ROO in the captured image 111 for each overlap region ROO. For example, the overlap detection unit 123 stores the derived position information 114 in the storage unit. For example, the overlap detection unit 123 stores the derived position information 114 in the storage unit in association with the overlap region ROO. The overlap region ROO is, for example, a rectangular region having the same size as or smaller than the smallest region of interest ROI among two or more regions of interest ROI that overlap each other. The position information 114 is composed of, for example, the coordinates (Xb, Yb) of the upper left end of the overlapping region ROO, the length of the overlapping region ROO in the X-axis direction, and the length of the overlapping region ROO in the Y-axis direction. The length of the overlapping region ROO in the X-axis direction is, for example, the physical region length XLb. The length of the overlapping region ROO in the Y-axis direction is, for example, the physical region length YLb. The position information 114 may include coordinates of a position different from the upper left end of the region of interest ROI.

[0025] The priority setting unit 124 assigns a priority 115 to each region of interest ROI in the captured image 111. For example, the priority setting unit 124 stores the assigned priority 115 in the storage unit. For example, the priority setting unit 124 associates the assigned priority 115 with the region of interest ROI and stores it in the storage unit. The priority setting unit 124 may assign the priority 115 to each region of interest ROI separately from the region number assigned to each region of interest ROI, or may use the region number assigned to each region of interest ROI instead of the priority 115. For example, the priority setting unit 124 may associate the priority 115 with the region of interest ROI and store it in the storage unit, or may associate the region number assigned to each region of interest ROI with the region of interest ROI and store it in the storage unit.

[0026] The priority 115 is an identifier of each region of interest ROI, and is discrimination information that can discriminate which of the multiple regions of interest ROI in the captured image 111 has had an overlapping region ROO omitted. For example, the priority setting unit 124 assigns 1 as the priority 115 to one of two regions of interest ROIs each including an overlapping region ROO, and assigns 2 as the priority 115 to the other region of interest ROI. In this case, when creating a transmission image 116 described later, the overlapping region ROO is omitted from the region of interest ROI having the larger priority 115 value. Note that the priority setting unit 124 may assign the same number as the region number assigned to each region of interest ROI as the priority 115 to the region of interest ROI. For example, the priority setting unit 124 associates the priority 115 assigned to each region of interest ROI with the ROI image 112 and stores it in the storage unit.

[0027] The encoding unit 125 encodes each transmission image 116 to generate compressed image data 120A. For example, the encoding unit 125 compresses each transmission image 116 using a compression format conforming to the JPEG standard as the format of the compressed image data 120A. The encoding unit 125 generates each transmission image 116 before performing the compression process. The encoding unit 125 generates a plurality of transmission images 116 in which the images 118 are omitted from the plurality of ROI images 112 obtained from the captured image 111 so that the images 118 of the overlapping regions ROO are not included in the plurality of ROI images 112 obtained from the captured image 111 in a duplicated manner.

[0028] The encoding unit 125 determines which of the multiple ROI images 112 the image 118 is to be omitted from, based on the priority 115 given to each region of interest ROI, for example. The encoding unit 125 may determine which of the multiple ROI images 112 the image 118 is to be omitted from, by using, for example, the region number given to each region of interest ROI as the priority 115. The encoding unit 125 determines the ROI image 112 specified as above from which the image 118 is omitted as the transmission image 116 (for example, the transmission image 116a2 in FIG. 3). For the ROI image 112 that does not include the overlapping region ROO or the ROI image 112 from which the image 118 is not to be omitted by the above determination, the encoding unit 125 determines the ROI image 112 itself as the transmission image 116 (for example, the transmission image 116a1 in FIG. 3).

[0029] The image processing control unit 126 generates the ROI information 120B and the frame information 120C and transmits them to the transmission unit 140. The ROI information 120B includes, for example, each piece of position information 113. The ROI information 120B further includes, for example, at least one of the data type of each region of interest ROI, the number of regions of interest ROI included in the captured image 111, the region number (or priority 115) of each region of interest ROI, the data length of each region of interest ROI, and the image format of each region of interest ROI. The frame information 120C includes, for example, the number of a virtual channel assigned to each frame, the data type of each region of interest ROI, the payload length for each line, and the like. The data type includes, for example, YUV data, RGB data, or RAW data. The data type further includes, for example, data in ROI format or data in normal format. The payload length is, for example, the number of pixels included in the payload of a long packet, for example, the number of pixels for each region of interest ROI. Here, the payload refers to the main data (application data) transmitted between the video transmitting device 100 and the video receiving device 200. The long packet refers to a packet placed between the packet header PH and the packet footer PF.

[0030] The transmitting unit 140 is a circuit that generates and transmits transmission data 147A based on various data (120A, 120B, 120C, 130A) input from the image processing units 120 and 130. The transmitting unit 140 transmits ROI information 120B for each region of interest ROI in the captured image 111 as embedded data (EmbeddedData). When a control signal instructing to cut out an ROI is input from the video receiving device 200 via the camera control interface CCI, the transmitting unit 140 further transmits image data of each region of interest ROI (compressed image data 120A) as payload data (PayloadData) of a long packet. At this time, the transmitting unit 140 transmits the image data of each region of interest ROI (compressed image data 120A) via a common virtual channel. The transmitting unit 140 also transmits image data (compressed image data 120A) of each region of interest ROI by an image data frame, and transmits ROI information 120B for each region of interest ROI by the header of the image data frame. The transmitting unit 140 also transmits normal image data (compressed image data 130A) by payload data of a long packet when a control signal instructing output of a normal image is input from the video receiving device 200 via the camera control interface CCI.

[0031] The transmission unit 140 has, for example, a LINK control unit 141, an ECC generation unit 142, a PH generation unit 143, an EBD buffer 144, an ROI data buffer 145, a normal image data buffer 146, and a synthesis unit 147. The LINK control unit 141, the ECC generation unit 142, the PH generation unit 143, the EBD buffer 144, and the ROI data buffer 145 output to the synthesis unit 147 when a control signal instructing cutting out an ROI is input from the video receiving device 200 via the camera control interface CCI. The normal image data buffer 146 outputs to the synthesis unit 147 when a control signal instructing output of a normal image is input from the video receiving device 200 via the camera control interface CCI.

[0032] The ROI data buffer 145 may also serve as the normal image data buffer 146. In this case, the transmission unit 140 may have a selector between the output terminals of the ROI data buffer 145 and the input terminal of the synthesis unit 147, which selects either the output of the ROI data buffer 145 or the output of the ROI data buffer 145.

[0033] The LINK control unit 141 outputs, for example, the frame information 120C to the ECC generation unit 142 and the PH generation unit 143 for each line. The ECC generation unit 142 generates an error correction code for one line based on data of the frame information 120C (for example, the number of the virtual channel, the data type of each region of interest ROI, the payload length for each line, etc.). The ECC generation unit 142 outputs, for example, the generated error correction code to the PH generation unit 143. The PH generation unit 143 generates a packet header PH for each line using, for example, the frame information 120C and the error correction code generated by the ECC generation unit 142. At this time, the packet header PH is, for example, a packet header of payload data of a long packet as shown in FIG. 4. This packet header PH includes, for example, DI, WC, and ECC. WC is an area for indicating the end of the packet to the video receiving device 200 by the number of words. WC includes, for example, the payload length, and includes, for example, the number of pixels for each region of interest ROI. The ECC is an area for storing values ​​for correcting bit errors. The ECC includes an error correction code. The DI is an area for storing a data identifier. The DI includes a VC (Virtual Channel) number and DataType (data type of each region of interest ROI). The VC (Virtual Channel) is a concept introduced for packet flow control, and is a mechanism for supporting multiple independent data streams sharing the same link. The PH generation unit 143 outputs the generated packet header PH to the synthesis unit 147.

[0034] The EBD buffer 144 temporarily stores the ROI information 120B and outputs the ROI information 120B as embedded data to the synthesis unit 147 at a predetermined timing. The embedded data refers to additional information that can be embedded in the header or footer of an image data frame (see FIG. 5 described later). The embedded data includes, for example, the ROI information 120B.

[0035] The ROI data buffer 145 temporarily stores the compressed image data 120A, and outputs the compressed image data 120A to the synthesis unit 147 as payload data of a long packet at a predetermined timing. When a control signal instructing to cut out an ROI is input from the video receiving device 200 via the camera control interface CCI, the ROI data buffer 145 outputs the compressed image data 120A to the synthesis unit 147 as payload data of a long packet. The normal image data buffer 146 temporarily stores the compressed image data 130A, and outputs the compressed image data 130A to the synthesis unit 147 as payload data of a long packet at a predetermined timing. When a control signal instructing to output a normal image is input from the video receiving device 200 via the camera control interface CCI, the normal image data buffer 146 outputs the compressed image data 130A to the synthesis unit 147 as payload data of a long packet.

[0036] When a control signal instructing output of a normal image is input from the video receiving device 200 via the camera control interface CCI, the synthesis unit 147 generates transmission data 147A based on the input data (compressed image data 130A). The synthesis unit 147 outputs the generated transmission data 147A to the video receiving device 200 via the data lane DL. On the other hand, when a control signal instructing cut-out of an ROI is input from the video receiving device 200 via the camera control interface CCI, the synthesis unit 147 generates transmission data 147A based on the input various data (packet header PH, ROI information 120B, and compressed image data 120A). The synthesis unit 147 outputs the generated transmission data 147A to the video receiving device 200 via the data lane DL. That is, the synthesis unit 147 includes DataType (data type of each region of interest ROI) in the packet header PH of the payload data of the long packet and sends it out. Moreover, the synthesis unit 147 outputs the image data (compressed image data 120A) of each region of interest ROI through a common virtual channel.

[0037] The transmission data 147A is configured by an image data frame as shown in FIG. 5, for example. The image data frame usually has a header area, a packet area, and a footer area. In FIG. 5, the footer area is omitted for convenience. The frame header area R1 of the transmission data 147A includes embedded data. At this time, the embedded data includes ROI information 120B. In FIG. 5, the packet area R2 of the transmission data 147A includes payload data of a long packet for each line, and further includes a packet header PH and a packet footer PF at a position where the payload data of the long packet is sandwiched. Furthermore, a low power mode LP is included at a position where the packet header PH and the packet footer PF are sandwiched.

[0038] At this time, the packet header PH includes, for example, DI, WC, and ECC. The WC includes, for example, a payload length, and includes, for example, the number of pixels for each region of interest ROI. The ECC includes an error correction code. The DI includes VC (a virtual channel number) and DataType (a data type for each region of interest ROI). In this embodiment, the VC of each line is assigned a common virtual channel number. In addition, in FIG. 5, the packet area R2 of the transmission data 147A includes compressed image data 147B. The compressed image data 147B is composed of one compressed image data 120A or a plurality of compressed image data 120A. Here, in FIG. 5, the packet group close to the packet header PH includes, for example, the compressed image data 120A (120A1) of the transmission image 116a1 in FIG. 3, and the packet group away from the packet header PH includes, for example, the compressed image data 120A (120A2) of the transmission image 116a2 in FIG. 3. These two compressed image data 120A1 and 120A2 constitute the compressed image data 147B. The payload data of the long packet for each line contains pixel data for one line in the compressed image data 147B.

[0039] Fig. 6 shows an example of the configuration of the transmission data 147A. The transmission data 147A includes, for example, a frame header area R1 and a packet area R2. Note that Fig. 6 shows a detailed example of the contents of the frame header area R1. Also, Fig. 6 omits the low power mode LP.

[0040] The frame header region R1 includes, for example, a frame number F1 as an identifier of the transmission data 147A. The frame header region R1 includes information about the compressed image data 147B included in the packet region R2. The frame header region R1 includes, for example, the number of compressed image data 120A included in the compressed image data 147B (the number of ROIs) and information about the ROI image 112 corresponding to each compressed image data 120A included in the compressed image data 147B (ROI information 120B).

[0041] For example, in the packet region R2 of the transmission data 147A, the synthesis unit 147 arranges the compressed image data 147B by dividing it into pixel rows of the compressed image data 120A. Therefore, the packet region R2 of the transmission data 147A does not include overlapping compressed image data corresponding to the image 118 in the overlapping region ROO. Also, for example, the synthesis unit 147 omits pixel rows of the captured image 111 that do not correspond to each transmission image 116 in the packet region R2 of the transmission data 147A. Therefore, the packet region R2 of the transmission data 147A does not include pixel rows of the captured image 111 that do not correspond to each transmission image 116. Note that in the packet region R2 of FIG. 6, the area surrounded by a dashed line corresponds to the compressed image data of the image 118 in the overlapping region ROO.

[0042] The boundary between the packet group closer to the packet header PH (e.g., 1(n) in FIG. 6) and the packet group farther from the packet header PH (e.g., 2(1) in FIG. 6) is specified by the physical region length XLa1 of the ROI image 112 corresponding to the compressed image data of the packet group closer to the packet header PH (e.g., 1(n) in FIG. 6). In the compressed image data corresponding to the image 118 of the overlapping region ROO included in the packet group closer to the packet header PH (e.g., 1(n) in FIG. 6), the starting position of the packet is specified by the physical region length XLa2 of the ROI image 112 corresponding to the packet group farther from the packet header PH (e.g., 2(1) in FIG. 6).

[0043] For example, when generating payload data of a long packet for each line in the packet region R2 of the transmission data 147A, the synthesis unit 147 may include, in the payload data of the long packet, the ROI information 120B, for example, as shown in FIG. 7, in addition to pixel data for one line in the compressed image data 147B. That is, the synthesis unit 147 may include the ROI information 120B in the payload data of the long packet and transmit it. In this case, the ROI information 120B includes at least one of the number of regions of interest ROI (number of ROIs) included in the captured image 111, the region number (or priority 115) of each region of interest ROI, the data length of each region of interest ROI, and the image format of each region of interest ROI, for example, as shown in FIG. 7(A) to FIG. 7(K). It is preferable that the ROI information 120B is arranged at the end of the payload data of the long packet on the packet header PH side (i.e., at the beginning of the payload data of the long packet).

[0044] (Video receiving device 200) Next, the video receiving device 200 will be described. FIG. 8 shows an example of the configuration of the video receiving device 200. FIG. 9 shows an example of a procedure for generating an ROI image 223A in the video receiving device 200. The video receiving device 200 is a device that receives a signal in a standard common to the video transmitting device 100 (for example, the MIP CSI-2 standard, the MIP CSI-3 standard, or the MIP CDI standard). The video receiving device 200 has, for example, a receiving unit 210 and an information processing unit 220. The receiving unit 210 is a circuit that receives the transmission data 147A output from the video transmitting device 100 via the data lane DL, performs a predetermined process on the received transmission data 147A, thereby generating various data (214A, 215A, 215B) and outputting the data to the information processing unit 220. The information processing unit 220 is a circuit that generates an ROI image 223A based on various data (214A, 215A) received from the receiving unit 210, and generates a normal image 224A based on data (215B) received from the receiving unit 210.

[0045] The receiving unit 210 includes, for example, a header separation unit 211, a header interpretation unit 212, a payload separation unit 213, an EBD interpretation unit 214, and an ROI data separation unit 215.

[0046] The header separation unit 211 receives the transmission data 147A from the video transmission device 100 via the data lane DL. That is, the header separation unit 211 receives the transmission data 147A including the ROI information 120B for each region of interest ROI in the captured image 111 in the embedded data and including the image data (compressed image data 120A) of each region of interest ROI in the payload data of the long packet. The header separation unit 211 separates the received transmission data 147A into a frame header region R1 and a packet region R2. The header interpretation unit 212 specifies the position of the payload data of the long packet included in the packet region R2 based on the data (specifically, the embedded data) included in the frame header region R1. The payload separation unit 213 separates the payload data of the long packet included in the packet region R2 from the packet region R2 based on the position of the payload data of the long packet specified by the header interpretation unit 212.

[0047] The EBD interpretation unit 214 outputs the embedded data as EBD data 214A to the information processing unit 220. The EBD interpretation unit 214 further determines, from the data type included in the embedded data, whether the image data included in the payload data of the long packet is compressed image data 120A of the ROI image data 116 or compressed image data 130A of normal image data. The EBD interpretation unit 214 outputs the determination result to the ROI data separation unit 215.

[0048] When the image data included in the payload data of the long packet is compressed image data 120A of the ROI image data 116, the ROI data separation unit 215 outputs the payload data of the long packet as payload data 215A to the information processing unit 220 (specifically, the ROI decoding unit 222). When the image data included in the payload data is compressed image data 130A of normal image data, the ROI data separation unit 215 outputs the payload data of the long packet as payload data 215B to the information processing unit 220 (specifically, the normal image decoding unit 224). When the payload data of the long packet includes ROI information 120B, the payload data 215A includes the ROI information 120B and one line of pixel data of the compressed image data 147B.

[0049] The information processing unit 220 extracts the ROI information 120B from the embedded data included in the EBD data 214A. The information processing unit 220 extracts an image of each region of interest ROI (ROI image 112) in the captured image 111 from the payload data of the long packet included in the transmission data 147A received by the receiving unit 210 based on the ROI information 120B extracted by the information extraction unit 221. The information processing unit 220 has, for example, the information extraction unit 221, an ROI decoding unit 222, an ROI image generation unit 223, and a normal image decoding unit 224.

[0050] The normal image decoding unit 224 decodes the payload data 215B to generate a normal image 224A. The ROI decoding unit 222 decodes the compressed image data 147B included in the payload data 215A to generate image data 222A. This image data 222A is composed of one or more transmission images 116.

[0051] The information extraction unit 221 extracts the ROI information 120B from the embedded data included in the EBD data 214A. For example, the information extraction unit 221 extracts the number of attention regions ROI included in the captured image 111, the area number (or priority 115) of each attention region ROI, the data length of each attention region ROI, and the image format of each attention region ROI from the embedded data included in the EBD data 214A. That is, the transmission data 147A includes the area number (or priority 115) of the attention region ROI corresponding to each transmission image 116 as discrimination information capable of discriminating which of the multiple transmission images 116 obtained from the transmission data 147A has omitted the image 118 of the overlapping region ROO.

[0052] Based on the ROI information 120B obtained by the information extraction section 221, the ROI image generation section 223 detects an overlapping region ROO where two or more regions of interest ROI overlap each other.

[0053] The information extraction unit 221 extracts, for example, from the embedded data included in the EBD data 214A, the coordinates (for example, the upper left coordinates (Xa1, Ya1)), length (for example, the physical region length XLa1, YLa1), and region number 1 (or priority 115 (=1)) of the region of interest ROI corresponding to the ROI image 112a1. The information extraction unit 221 further extracts, for example, from the embedded data included in the EBD data 214A, the coordinates (for example, the upper left coordinates (Xa2, Ya2)), length (for example, the physical region length XLa2, YLa2), and region number 2 (or priority 115 (=2)) of the region of interest ROI corresponding to the ROI image 112a2.

[0054] At this time, the ROI image generating unit 223 derives the position information 114 of the overlapping region ROO based on the extracted information (hereinafter referred to as "extracted information 221A"). The ROI image generating unit 223 derives, for example, the coordinates (e.g., the upper left corner coordinate (Xb1, Yb1)) and length (e.g., the physical region length XLb1, YLb1) of the overlapping region ROO as the position information 114 of the overlapping region ROO.

[0055] The ROI image generating unit 223 may obtain the ROI information 120B from the payload data 215A instead of obtaining the ROI information 120B from the embedded data included in the EBD data 214A. In this case, the ROI image generating unit 223 may detect an overlapping region ROO where two or more regions of interest ROI overlap each other based on the ROI information 120B included in the payload data 215A. The ROI image generating unit 223 may also extract the extraction information 221A from the ROI information 120B included in the payload data 215A, and may derive the position information 114 of the overlapping region ROO based on the extraction information 221A extracted in this way.

[0056] The ROI image generating unit 223 further generates images of the regions of interest ROI (ROI images 112a1, 112a2) in the captured image 111 based on the image data 222A, the extraction information 221A, and the position information 114 of the overlapping region ROO. The ROI image generating unit 223 outputs the generated images as ROI image 223A.

[0057] [procedure] Next, an example of a procedure for data transmission in the video transmission system 1 will be described with reference to FIGS.

[0058] First, the imaging unit 110 outputs a captured image 111 (digital image data) obtained by imaging to the image processing unit 120. The ROI cutout unit 121 specifies two regions of interest ROI1 and ROI2 included in the captured image 111 input from the imaging unit 110. The ROI cutout unit 121 cuts out images of the regions of interest ROI1 and ROI2 (ROI images 112a1 and 112a2) from the captured image 111. The ROI cutout unit 121 assigns region number 1 as an identifier to the region of interest ROI1, and assigns region number 2 as an identifier to the region of interest ROI2.

[0059] The ROI analysis unit 122 derives position information 113 of the region of interest ROI in the captured image 111 for each region of interest ROI. Based on the region of interest ROI1, the ROI analysis unit 122 derives the top left coordinate (Xa1, Ya1) of the region of interest ROI1, the length of the region of interest ROI1 in the X-axis direction (XLa1), and the length of the region of interest ROI1 in the Y-axis direction (YLa1). Based on the region of interest ROI2, the ROI analysis unit 122 derives the top left coordinate (Xa2, Ya2) of the region of interest ROI2, the length of the region of interest ROI2 in the X-axis direction (XLa2), and the length of the region of interest ROI2 in the Y-axis direction (YLa2).

[0060] The overlap detection unit 123 detects an overlapping region ROO where the two regions of interest ROI1, ROI2 overlap with each other, based on position information 113 of the two regions of interest ROI1, ROI2 in the captured image 111. That is, the overlap detection unit 123 derives position information 114 of the overlapping region ROO in the captured image 111. The overlap detection unit 123 derives the upper left corner coordinates (Xb1, Yb1) of the overlapping region ROO, the length in the X-axis direction of the overlapping region ROO (XLb1), and the length in the Y-axis direction of the overlapping region ROO (YLb1) as the position information 114 of the overlapping region ROO in the captured image 111.

[0061] Of the two regions of interest ROI1 and ROI2, the priority setting unit 124 assigns a priority 115 of 1 to the region of interest ROI1 and assigns a priority 115 of 2 to the other region of interest ROI2.

[0062] The encoding unit 125 generates two transmitted images 116a1, 116a2 by omitting the image 118 from the two ROI images 112a1, 112a2 obtained from the captured image 111 so that the image 118 of the overlap region ROO is not overlapped in the two regions of interest ROI1, ROI2.

[0063] The encoding unit 125 determines which of the two ROI images 112a1 and 112a2 the image 118 is to be omitted from, based on the region numbers (or priority 115) of the two regions of interest ROI1 and ROI2. The encoding unit 125 omits the image 118 from the ROI image 112a2 corresponding to the region of interest ROI2, which has the larger region number (or priority 115), of the two regions of interest ROI1 and ROI2, thereby generating a transmission image 116a2. For the ROI image 112a1 corresponding to the region of interest ROI1, which has the smaller region number (or priority 115) of the two regions of interest ROI1 and ROI2, the encoding unit 125 sets the ROI image 112a1 itself as the transmission image 116a1.

[0064] The image processing control unit 126 generates ROI information 120B and frame information 120C and transmits them to the transmission unit 140. The transmission unit 140 generates transmission data 147A based on various data (120A, 120B, 120C, 130A) input from the image processing units 120 and 130. The transmission unit 140 sends the generated transmission data 147A to the video receiving device 200 via the data lane DL.

[0065] The receiving unit 210 receives the transmission data 147A output from the video transmitting device 100 via the data lane DL. The receiving unit 210 performs a predetermined process on the received transmission data 147A to generate EBD data 214A and payload data 215A, and outputs them to the information processing unit 220.

[0066] The information extraction unit 221 extracts the ROI information 120B from the embedded data included in the EBD data 214A. The information extraction unit 221 extracts the coordinates (e.g., upper left coordinates (Xa1, Ya1)), length (e.g., physical region length XLa1, YLa1), and region number 1 (or priority 115 (=1)) of the region of interest ROI corresponding to the ROI image 112a1 from the embedded data included in the EBD data 214A. The information extraction unit 221 further extracts the coordinates (e.g., upper left coordinates (Xa2, Ya2)), length (e.g., physical region length XLa2, YLa2), and region number 2 (or priority 115 (=2)) of the region of interest ROI corresponding to the ROI image 112a2. The ROI decoding unit 222 decodes the compressed image data 147B included in the payload data 215A to generate image data 222A.

[0067] The ROI image generating unit 223 derives the position information 114 of the overlapping region ROO based on the extracted information (extraction information 221A). The ROI image generating unit 223 derives, for example, the coordinates (e.g., upper left corner coordinate (Xb1, Yb1)) and length (e.g., physical region length XLb1, YLb1) of the overlapping region ROO as the position information 114 of the overlapping region ROO. The ROI image generating unit 223 further generates images of the regions of interest ROI (ROI images 112a1, 112a2) in the captured image 111 based on the image data 222A, the extraction information 221A, and the position information 114 of the overlapping region ROO.

[0068] [effect] Next, effects of the video transmission system 1 according to the present embodiment will be described.

[0069] In recent years, there has been an increase in applications that require the transmission of large amounts of data. This puts a heavy load on the transmission system, and in the worst case scenario, the transmission system may crash, making it impossible to transmit data.

[0070] Conventionally, in order to avoid a breakdown in the transmission system, for example, instead of transmitting all captured images, the object being photographed is identified and only a portion of the image extracted from the identified object is transmitted.

[0071] By the way, MIPICSI-2 is sometimes used as a method for transmitting ROI from an image sensor to an application processor. However, when trying to transmit ROI using this method, it may not be easy to transmit the ROI due to various constraints.

[0072] On the other hand, in this embodiment, ROI information 120B for each region of interest ROI in captured image 111 is sent as embedded data, and image data of each region of interest ROI is sent as payload data of a long packet. This allows the device (video receiving device 200) that receives transmission data 147A sent from video transmitting device 100 to easily extract image data (ROI image 112) of each region of interest ROI from transmission data 147A. As a result, the region of interest ROI can be transmitted even under various constraints.

[0073] In this embodiment, image data (compressed image data 120A) of each region of interest ROI is sent through a common virtual channel. This allows multiple ROI images 112 to be sent in the same packet, making it unnecessary to enter LP mode while sending multiple ROI images 112, and thus achieving high transmission efficiency.

[0074] In this embodiment, the data type of each region of interest ROI is included in the packet header PH of the payload data of the long packet and sent out. This allows the data type of each region of interest ROI to be obtained simply by accessing the packet header PH of the payload data of the long packet without having to access the embedded data. This allows the processing speed in the video receiving device 200 to be increased, resulting in high transmission efficiency.

[0075] In addition, in this embodiment, when the ROI information 120B is included in the payload data of a long packet and sent, the ROI information 120B can be obtained by only accessing the payload data of the long packet without accessing the embedded data. This can increase the processing speed in the video receiving device 200, thereby achieving high transmission efficiency.

[0076] In this embodiment, ROI information 120B for each region of interest ROI is extracted from embedded data included in the transmission data 147A, and an image of each region of interest ROI (ROI image 112) is extracted from payload data of a long packet included in the transmission data 147A based on the extracted ROI information 120B. This makes it possible to easily extract an image of each region of interest ROI (ROI image 112) from the transmission data 147A. As a result, it is possible to transmit the region of interest ROI even under various constraints.

[0077] 2. Prerequisite technology 2 of this disclosure: A technique for transmitting a region of interest (ROI) of a part (having a non-rectangular shape) cut out from a captured image will be described with reference to Figs. 1 to 9 and Figs. 10 to 12. That is, a technique for transmitting and receiving an image of an object of a capture target having a shape other than a square (rectangular) shape will be described. Fig. 10 is a diagram showing a schematic area in which an object identified in a captured image 111 is located. Note that, for ease of understanding, Fig. 10 shows a schematic area in which a captured image 111 captured in an imaging area composed of imaging elements of 15 rows x 23 columns. Fig. 11 is a diagram showing an example of an ROI area set for an identified object.

[0078] In the base technology 2, as in the base technology 1, a case will be described in which a control signal instructing to cut out an ROI is input from the video receiving device 200 to the video transmitting device 100 via the camera control interface CCI, and a predetermined process is performed on the captured image 111 input from the imaging unit 110. However, the base technology 2 can also be applied to a case in which the video transmitting device 100, i.e., the transmitting side, instructs the coordinates of the ROI cut-out. In this case, the transmitting side is configured to receive information such as "person" or "object" to be acquired in the ROI sent from the receiving side, and determine and instruct the coordinates of the cut-out.

[0079] A control signal instructing to cut out the ROI is input from the video receiving device 200 via the camera control interface CCI. As a result, as shown in FIG. 10, the ROI cutout unit 121 specifies four objects 1 to 4 of the shooting target included in the captured image 111. For example, the object 1 has a rectangular shape occupying a part of the upper left region in the captured image 111. For example, the object 2 occupies a part of the right side region of the object 1 in the captured image 111, and has a shape in which both upper corners and a part of the bottom side are missing. For example, the object 3 occupies a part of the lower region of the object 2 in the captured image 111, and has a rectangular shape in which four corners are missing. For example, the object 4 occupies a part of the lower region of the object 3 in the captured image 111, and has a rectangular shape in which both upper corners are missing. The objects 3 and 4 partially overlap each other.

[0080] As shown in FIG. 11, the ROI cutout unit 121 (see FIG. 2) sets the smallest rectangles including the identified objects 1 to 4 as regions of interest ROI1 to ROI4. The ROI cutout unit 121 sets a region of interest ROI1 for object 1 and cuts out an ROI image 112a1. The ROI cutout unit 121 also sets a region of interest ROI2 for object 2 and cuts out an ROI image 112a2. The ROI cutout unit 121 also sets a region of interest ROI3 for object 3 and cuts out an ROI image 112a3. The ROI cutout unit 121 also sets a region of interest ROI4 for object 4 and cuts out an ROI image 112a4.

[0081] The ROI cutout unit 121 associates the region of interest ROI1 with the region number "1" assigned to the region of interest ROI1 and stores them in the storage unit. The ROI cutout unit 121 associates the region of interest ROI2 with the region number "2" assigned to the region of interest ROI2 and stores them in the storage unit. The ROI cutout unit 121 associates the region of interest ROI3 with the region number "3" assigned to the region of interest ROI3 and stores them in the storage unit. The ROI cutout unit 121 associates the region of interest ROI4 with the region number "4" assigned to the region of interest ROI4 and stores them in the storage unit.

[0082] The ROI analysis unit 122 (see FIG. 2) derives position information of each of the regions of interest ROI1 to ROI4. The ROI analysis unit 122 derives, for example, a physical region length XLa1 in the X-axis direction and a physical region length YLa1 in the Y-axis direction as the position information of the region of interest ROI1. The ROI analysis unit 122 derives, for example, a physical region length XLa2 in the X-axis direction and a physical region length YLa2 in the Y-axis direction as the position information of the region of interest ROI2. The ROI analysis unit 122 derives, for example, a physical region length XLa3 in the X-axis direction and a physical region length YLa3 in the Y-axis direction as the position information of the region of interest ROI3. The ROI analysis unit 122 derives, for example, a physical region length XLa4 in the X-axis direction and a physical region length YLa4 in the Y-axis direction as the position information of the region of interest ROI4. The ROI analysis unit 122 may further derive, for each region of interest ROI, as the position information 113, for example, an output region length XLc of the region of interest ROI in the X-axis direction and an output region length YLc of the region of interest ROI in the Y-axis direction.

[0083] The ROI analysis unit 122 derives the length of each of the regions of interest ROI in the X-axis direction and the Y-axis direction, thereby deriving the size and total data amount of each of the regions of interest ROI1 to ROI4 as information for the subsequent stage. This allows the video receiving device 200, which corresponds to the subsequent stage, to secure a memory area.

[0084] The ROI analysis unit 122 is configured to derive position information of the ROI images 112a1-112a4, instead of position information of the region of interest ROI, when the shapes of the object to be photographed and the region of interest do not match. The ROI analysis unit 122 derives the left end coordinates (xn,yn) of each row and the physical region length XLn in the X-axis direction as position information of the ROI images 112a1-112a4. When the ROI images are separated, such as the second row of the ROI image 112a2, the ROI analysis unit 122 derives position information for each of the separated parts. The ROI analysis unit 122 associates the region numbers of the regions of interest ROI1-ROI4 with the position information of the ROI images 112a1-112a4 and stores them in the storage unit.

[0085] In addition, the ROI analysis unit 122 may derive, for example, sensing information, exposure information, gain information, AD word length, image format, etc. in addition to the position information for each of the regions of interest ROI1 to ROI4, and store them in the storage unit in association with the region number.

[0086] When the object to be photographed is rectangular, the overlap detection unit 123 (see FIG. 2) derives an overlap region between ROI images, not an overlap region between attention regions, as an overlap region. As shown in FIG. 11, the overlap detection unit 123 derives an overlap region ROO as an overlap region between the ROI image 112a3 and the ROI image 123a4. The overlap detection unit 123 stores the derived overlap region ROO in the storage unit in association with the position information of the attention regions ROI3 and ROI4.

[0087] The priority setting unit 124 (see FIG. 2) assigns a priority of "1" to the region of interest ROI1, and stores the priority 1 in association with the region of interest ROI1 in the storage unit. The priority setting unit 124 assigns a priority of "2" lower than the priority "1" to the region of interest ROI2, and stores the priority 2 in association with the region of interest ROI2 in the storage unit. The priority setting unit 124 assigns a priority of "3" lower than the priority "2" to the region of interest ROI3, and stores the priority 3 in association with the region of interest ROI3 in the storage unit. The priority setting unit 124 assigns a priority of "4" lower than the priority "3" to the region of interest ROI4, and stores the priority 4 in association with the region of interest ROI4 in the storage unit.

[0088] The encoding unit 125 (see FIG. 2) generates a transmission image for each of the ROI images 112a1 to 112a4. Since the region of interest ROI4 has a lower priority than the region of interest ROI3, the encoding unit 125 generates a transmission image by omitting the overlapping region ROO from the ROI image 112a4.

[0089] The image processing control unit 126 (see FIG. 2) generates ROI information and frame information, and transmits them to the transmission unit 140 (see FIG. 2). The ROI information includes, for example, position information of each of the ROI images 112a1-112a4. The ROI information also includes information similar to that in the case where the object to be photographed is rectangular (for example, the data type of each of the regions of interest ROI1-ROI4, the number of regions of interest ROI1-ROI4 included in the captured image 111, region numbers and priorities of the regions of interest ROI1-ROI4, etc.). The frame information includes, for example, the data type of the regions of interest ROI1-ROI4, etc., similar to that in the case where the object to be photographed is rectangular (for example, the data type of the regions of interest ROI1-ROI4, etc.).

[0090] The LINK control unit 141 provided in the transmission unit 140 (see FIG. 2) outputs the frame information and ROI information input from the image processing control unit 126 to the ECC generation unit 142 and the PH generation unit 143 (see FIG. 2 for both). For example, the ECC generation unit 142 generates an error correction code for one line based on data of the line in the frame information (for example, the virtual channel number, the data type of each of the regions of interest ROI1 to ROI4, the payload length for each line, etc.). For example, the ECC generation unit 142 outputs the generated error correction code to the PH generation unit 143. For example, the PH generation unit 143 generates a packet header PH (see FIG. 4) for each line using the frame information and the error correction code generated by the ECC generation unit 142.

[0091] The EBD buffer 144 (see FIG. 2) temporarily stores the ROI information, and outputs the ROI information as embedded data to the synthesis unit 147 (see FIG. 2) at a predetermined timing.

[0092] The ROI data buffer 145 (see FIG. 2) temporarily stores the compressed image data input from the encoding unit 125, and outputs the compressed image data 120A to the synthesis unit 147 as payload data of a long packet when, for example, a control signal instructing extraction of an ROI is input from the video receiving device 200 via the camera control interface CCI.

[0093] When a control signal instructing to extract an ROI is input from the video receiving device 200 via the camera control interface CCI, the synthesis unit 147 generates transmission data 147A based on the various input data (packet header PH, ROI information, and compressed image data input from the encoding unit 125 via the ROI data buffer 145). The synthesis unit 147 outputs the generated transmission data 147A to the video receiving device 200 via the data lane DL. That is, the synthesis unit 147 includes the data type of each of the regions of interest ROI1 to ROI4 in the packet header PH of the payload data of the long packet and transmits it. Furthermore, the synthesis unit 147 transmits the image data (compressed image data) of each of the regions of interest ROI1 to ROI4 via a common virtual channel.

[0094] If the object to be photographed is not rectangular, the position information of the ROI images 112a1-112a4 is included in the packet header PH or the payload data of the long packet. The position information of the ROI images 112a1-112a4 is included in the packet header PH by the PH generation unit 143. On the other hand, the position information of the ROI images 112a1-112a4 is included in the payload data of the long packet by the synthesis unit 147.

[0095] Fig. 12 is a diagram showing a configuration example of transmission data 147A in which position information of ROI images 112a1-112a4 is included in payload data of a long packet. As shown in Fig. 12, the transmission data 147A includes, for example, a frame header area R1 and a packet area R2. Note that Fig. 12 shows an example of the contents of the frame header area R1 in detail. Also, Fig. 12 omits the low power mode LP.

[0096] The frame header region R1 includes, for example, a frame number F1 as an identifier of the transmission data 147A. The frame header region R1 includes information about the compressed image data included in the packet region R2. The frame header region R1 includes, for example, the number of compressed image data (number of ROIs) and information (ROI information) about each of the ROI images 112a1 to 112a4 corresponding to each compressed image data. The ROI information includes a region number, a physical region length, a rectangular output region size, a priority, exposure information, gain information, an AD word length, and an image format. The physical region length is the maximum length of the ROI image, and the rectangular output region size is the size of the region of interest ROI.

[0097] "Info" shown in FIG. 12 indicates area information stored in the payload of a long packet. The position information of the ROI images 112a1 to 112a4 is stored in, for example, "info". The position information of the ROI images 112a1 to 112a4 is stored in the beginning of the payload of a long packet. When the physical area length in the X-axis direction of each of the consecutive pixel rows constituting an ROI image is the same and the corresponding pixel rows do not include ROI images with different area numbers, the payload of a long packet including image data of the second and subsequent pixel rows among the corresponding pixel rows may not store the area information "info". In this example, the ROI image 112a1 has the same physical area length in the X-axis direction in the consecutive first to fourth pixel rows among all pixel rows, and the corresponding first to fourth pixel rows do not include ROI images with different area numbers. Therefore, the payload of each long packet including image data of the second to fourth pixel rows, which correspond to the second and subsequent rows among the first to fourth consecutive pixel rows constituting the ROI image 112a1, does not store the region information "info". In addition, in this example, the ROI image 112a4 has the same physical region length in the X-axis direction in the second and third consecutive pixel rows among all pixel rows, and the second and third pixel rows do not include ROI images with different region numbers. Therefore, the payload of a long packet including image data of the third pixel row, which corresponds to the second and subsequent rows among the second and third consecutive pixel rows constituting the ROI image 112a4, does not store the region information "info". Note that even if the physical region length in the X-axis direction is the same and the pixel rows do not include ROI images with different region numbers, the region information "info" may be stored in the payload of each row.

[0098] For example, the synthesis unit 147 arranges the compressed image data generated by compressing each of the ROI images 112a1 to 112a4 in the packet region R2 of the transmission data 147A by dividing them into pixel rows. "1" shown in FIG. 12 indicates the compressed image data of the ROI image 112a1 stored in the payload of the long packet. "2" shown in FIG. 12 indicates the compressed image data of the ROI image 112a2 stored in the payload of the long packet. "3" shown in FIG. 12 indicates the compressed image data of the ROI image 112a3. "4" shown in FIG. 12 indicates the compressed image data of the ROI image 112a4 stored in the payload of the long packet. Note that in FIG. 12, each compressed image data is shown separated for ease of understanding, but the data stored in the payload of the long packet is not separated. The packet region R2 of the transmission data 147A does not include the compressed image data 112b corresponding to the image of the overlap region ROO in a duplicated manner. Furthermore, for example, in the packet region R2 of the transmission data 147A, the synthesis unit 147 omits pixel rows that do not correspond to any of the transmission images in the captured image 111. Therefore, the packet region R2 of the transmission data 147A does not include pixel rows that do not correspond to any of the transmission images in the captured image 111.

[0099] Next, the operation of the video receiving device 200 when it receives the transmission data 147A will be described. The header separation unit 211 (see FIG. 8 for both) provided in the receiving unit 210 receives the transmission data 147A from the video transmitting device 100 via the data lane DL. That is, the header separation unit 211 receives the transmission data 147A including ROI information for the regions of interest ROI1 to ROI4 in the captured image 111 in embedded data and including image data (compressed image data) of the regions of interest ROI1 to ROI4 in payload data of a long packet. The header separation unit 211 separates the received transmission data 147A into a frame header region R1 and a packet region R2.

[0100] The header interpretation unit 212 (see FIG. 8) identifies the position of the payload data of the long packet included in the packet area R2 based on the data (specifically, embedded data) included in the frame header area R1.

[0101] Based on the position of the payload data of the long packet identified by the header interpretation unit 212, the payload separation unit 213 (see FIG. 8) separates the payload data of the long packet contained in the packet area R2 from the packet area R2.

[0102] The EBD interpretation unit 214 outputs the embedded data as EBD data to the information processing unit 220 (see FIG. 8). The EBD interpretation unit 214 further determines, from the data type included in the embedded data, whether the image data included in the payload data of the long packet is compressed image data of the ROI image data 116 or compressed image data of normal image data. The EBD interpretation unit 214 outputs the determination result to the ROI data separation unit 215 (see FIG. 8).

[0103] When image data included in the payload data of a long packet is input as ROI image data, the ROI data separation unit 215 outputs the payload data of the long packet as payload data to the information processing unit 220 (specifically, the ROI decoding unit 222 (see FIG. 8)). The payload data of the long packet including the ROI information includes the ROI information and pixel data for one line of the compressed image data.

[0104] The information extraction unit 221 (see FIG. 8) provided in the information processing unit 220 extracts the number of regions of interest ROI1-ROI4 (four in this example) included in the captured image 111, the region numbers 1-4 and priorities 1-4 of the regions of interest ROI1-ROI4, the data lengths of each of the regions of interest ROI1-ROI4, and the image formats of each of the regions of interest ROI1-ROI4, from the embedded data included in the EBD data input from the EBD interpretation unit 214. Furthermore, the information extraction unit 221 extracts position information of the ROI images 112a1-112a4 from the embedded data.

[0105] The ROI decoding unit 222 decodes the compressed image data 147B included in the payload data, extracts position information of the ROI images 112a1 to 112a4, and generates image data (composed of the transmitted image). For example, when payload data corresponding to the sixth pixel row is input, the ROI decoding unit 222 extracts one piece of position information of the ROI image 112a1 and two pieces of position information of the ROI image 112a2 from the payload data, and generates image data (transmitted images) of the ROI images 112a1 and 112b1 corresponding to the sixth pixel row.

[0106] When payload data corresponding to, for example, the 10th pixel row is input, the ROI decoding unit 222 extracts one piece of position information of the ROI image 112a3 and one piece of position information of the ROI image 112a4 from the payload data, and generates image data (transmitted images) of the ROI images 112a3 and 112b4, respectively.

[0107] The ROI image generation unit 223 (see FIG. 8) generates ROI images 112a1-112a4 of the regions of interest ROI1-ROI4 in the captured image 111 based on the ROI information obtained by the information extraction unit 221, the position information of the ROI images 112a1-112a4 extracted by the ROI decoding unit 222, and the transmitted image generated by the ROI decoding unit 222. For example, when the ROI image generation unit 223 receives one piece of position information of ROI image 112a1 and two pieces of position information of ROI image 112a2 extracted from payload data corresponding to the sixth pixel row, as well as their transmission images, it generates an ROI image 112a1 of five pixels extending in the X-axis direction, an ROI image 112a2 of four pixels extending in the X-axis direction at a position five pixels away from the ROI image 112a1, and an ROI image 112a2 of two pixels extending in the X-axis direction at a position two pixels away from the ROI image 112a2 (see FIG. 10).

[0108] Furthermore, the ROI image generating unit 223 detects an overlapping area ROO where the regions of interest ROI3 and ROI4 overlap, based on the ROI information obtained by the information extracting unit 221. The ROI image generating unit 223 generates an ROI image 112a3 of four pixels extending in the X-axis direction and an ROI image 112a4 of three pixels extending in the X-axis direction with one pixel overlapping the ROI image 112a3, based on the detected overlapping area ROO, the position information of the ROI images 112a3 and 112a4 extracted from the payload data corresponding to the tenth pixel row, and the transmitted image (see FIG. 10).

[0109] The ROI image generating unit 223 outputs the generated image as an ROI image to a downstream device (not shown).

[0110] In this way, the video transmitting device 100 and the video receiving device 200 can transmit and receive an ROI image even if the object to be photographed has a shape other than a rectangle.

[0111] 3. Principle of restoration process in the embodiment of the present disclosure: Next, the principle of the restoration process in an embodiment of the present disclosure will be described with reference to Figures 13 to 17. The restoration process in an embodiment of the present disclosure (hereinafter referred to as this embodiment) is a process of restoring image data of a region of interest (ROI) input from a video transmitting device to a video receiving device to a state that can be displayed on a specified display device. By executing the restoration process in the video receiving device and restoring the image data of the region of interest, for example, various processes related to the display of the image data of the region of interest (for example, writing to a frame memory, outputting the image data of the region of interest to a specified display device, etc.) can be efficiently performed.

[0112] 3-1. Transmission of image data of a region of interest in a video transmission device First, as a preliminary step of the restoration process, the process up to when image data of the region of interest is input to the video receiving device will be described. As described above, image data of the region of interest cut out from the captured image is sent from the video transmitting device as payload data. FIG. 13 is a diagram showing an example of a captured image obtained by the imaging section of the video transmitting device. In FIG. 13, the captured image α is shown in correspondence with pixel coordinates. In this example, multiple (three in this example) regions of interest (ROI0, ROI1, ROI2) are set in the captured image α. Hereinafter, the restoration process will be described using the regions of interest ROI0 to ROI2 shown in FIG. 13 as a specific example.

[0113] In this example, the coordinate system of the captured image α is an XY Cartesian coordinate system formed by the X-axis direction (horizontal direction) and the Y-axis direction (vertical direction) of the captured image α, with the pixel at the upper left end as the origin α_о. Also, as shown in FIG. 13, the coordinates of the lower right end (end point) α_e of the captured image α are (1759, 989). That is, the captured image α in this example is a rectangular image with a width (length in the X-axis direction) of 1760 pixels (Pixels) and a height (length in the Y-axis direction) of 990 pixels. In this example, it is assumed that three regions of interest (ROI0, ROI1, ROI2) set based on instructions in a control signal from a video receiving device, for example, are cut out of the captured image α. The regions of interest ROI0 to 2 are all rectangular regions. Also, the regions of interest ROI0 to 2 are arranged at a predetermined interval without overlapping with each other. Also, an area in the captured image α where no region of interest ROI is arranged is defined as an out-of-interest area OE.

[0114] FIG. 14 is a diagram showing the coordinates and image size of each of three regions of interest (ROI1, ROI2, ROI3) in the captured image α. In the coordinates of the regions of interest, the coordinate closest to the origin α_о of the captured image α is the top left coordinate (start coordinate), and the coordinate farthest from the origin α_о is the bottom right coordinate (end coordinate). The size of the region of interest includes an X-axis length indicating the width of the region of interest and a Y-axis length indicating the height of the region of interest. The arrangement state of the rectangular region of interest in the captured image, that is, the position in the captured image and the image size, can be obtained by using at least the start coordinate, the X-axis length, and the Y-axis length.

[0115] 14, in this example, the region of interest ROI0 is located in the captured image α at a position where the start point coordinates are start point X coordinate i0=520, start point Y coordinate j0=300, and the end point coordinates are end point X coordinate k0=919, end point Y coordinate l0=899. The region of interest ROI1 is located at a coordinate position where the start point coordinates are start point X coordinate i1=1040, start point Y coordinate j1=140, and the end point coordinates are end point X coordinate k1=1439, end point Y coordinate l1=739. The region of interest ROI2 is located at a coordinate position where the start point coordinates are start point X coordinate i2=100, start point Y coordinate j2=100, and the end point coordinates are end point X coordinate k2=499, end point Y coordinate l2=699. In this example, each of the regions of interest ROI0 to ROI2 has an X-axis length of 400 pixels and a Y-axis length of 600 pixels.

[0116] As shown in FIG. 13 and FIG. 14, in this example, the X coordinates of the start point coordinates of the attention regions ROI0-2 are in the order of the X coordinate i2 (100) of the attention region ROI2 closest to the origin α_о, followed by the start point X coordinate i0 (520) of the attention region ROI0, and the start point X coordinate i1 (1040) of ROI1. Therefore, the relative positions of the attention regions ROI0-2 in the X-axis direction in the captured image α are in the order of the attention regions ROI2, ROI0, and ROI1. Also, in this example, the Y coordinates of the start point coordinates of the attention regions ROI0-2 are in the order of the attention region ROI2 closest to the origin α_о, followed by the attention regions ROI1 and ROI0. Therefore, as shown in FIG. 13, the relative positions of the attention regions ROI0-2 in the Y-axis direction in the captured image α are in the order of the attention regions ROI2, ROI1, and ROI0.

[0117] In the video transmitting device, when a control signal instructing the cutting out of ROI0-2 shown in FIG. 13 is input from the video receiving device, for example, the cut-out positions for cutting out an image from the imaging area of ​​the imaging unit 31, that is, the start point coordinates and image size (length in the X-axis direction and Y-axis direction) of each of the regions of interest ROI0-2 are determined. The determined start point coordinates and image size are included in the ROI information as position information and size information, respectively. In other words, the ROI information includes position information and size information corresponding to a predetermined number of regions of interest (ROI) cut out from the captured image. The ROI information is set as embedded data in the frame header area of ​​the transmission data, and is sent out from the video transmitting device.

[0118] The ROI information set in the embedded data may be restricted by the specifications of the transmission standard. In this embodiment, the ROI information is generated with the start point coordinate and the image size as the position information and the size information, respectively, as the minimum data that can determine the position and size of the region of interest in the captured image in the video receiving device. In this embodiment, the end point coordinate indicating the lower right end of the region of interest is not included in the ROI information, but in FIG. 14, the end point coordinate is displayed together with the start point coordinate for ease of understanding. Note that the present disclosure may be configured to include the end point coordinate of the region of interest as the position information in the ROI information. When the region of interest is rectangular, the position information in the present disclosure may be the coordinate indicating any one of the four corners.

[0119] 15 is a diagram showing an example of payload data including image data of the regions of interest ROI0 to 2. The payload data is set in a packet area of ​​transmission data and transmitted from the video transmitting device. In this embodiment, image data of the region of interest is stored in units of pixel rows, that is, in units of Y-axis coordinates in a captured image, in each line of the payload data.

[0120] In this embodiment, the payload data storing the region of interest includes only image data of the region of interest in the captured image, and does not store image data of the non-interest region OE. Therefore, pixel rows consisting of only image data of the non-interest region OE in the captured image are not included in the payload data. Also, the image data of the region of interest in the payload data is stored in the payload data in a state where it is left-justified toward the payload header PH (corresponding to the "packet header PH" in the above-mentioned base technologies 1 and 2). As shown in FIG. 15, the image data of the region of interest in the captured image is stored in the payload data in a state where it is filled in the upper left direction, that is, toward the origin (in this example, the origin α_о) of the captured image. In this example, the image data of each of the regions of interest ROI0 to 2 corresponds to 800 pixel rows of the Y-axis coordinate in the captured image α. Therefore, in this example, the payload data storing the image data of each of the regions of interest ROI0 to 2 is composed of a total of 800 lines of lines PL1 to 800 in order from the beginning, and constitutes a data group storing 800 pixel rows of the Y-axis coordinate in the captured image α.

[0121] In this example, of the three regions of interest (ROI) in the captured image α, the region with the smallest Y coordinate of the starting point coordinate is the region of interest ROI2. Therefore, as shown in FIG. 15, the image data of the first row of the pixel rows of the region of interest ROI2 (in this example, image data ROI2(L1)) is stored left-justified in the line PL1, which is the first row of the payload data. Also, in this example, of the three regions of interest in the captured image α, the region with the largest Y coordinate of the starting point coordinate is the region of interest ROI0. Therefore, as shown in FIG. 15, the image data of the last row of the pixel rows of the region of interest ROI0 (in this example, image data ROI0(L600)) is stored left-justified in the line PL800, which is the last row of the payload data.

[0122] In this way, the image data of the regions of interest ROI0-2 in the captured image α in this example are stored in each line of the payload data, left-justified, starting from the image data of the region with the smallest Y coordinate value of the starting point coordinate (Y coordinates j0-2). For example, in the lines PL1-40 of the payload data shown in Fig. 15, image data ROI2(L1)-(L40) from the 1st pixel row (Y coordinate = 100) to the 140th pixel row (Y coordinate = 139), which corresponds to the Y coordinate j2 of the starting point coordinate of the region of interest ROI2, are stored left-justified.

[0123] As described above, image data of the region of interest is stored in each line of the payload data in units of Y-axis coordinates (pixel row units) in the captured image. The image data of the region of interest in units of pixel rows stored in the payload data is called ROI pixel row data. The ROI pixel row data is composed of a group of packets containing compressed image data of the region of interest. When multiple regions of interest are located at the same Y coordinate in the captured image, the ROI pixel row data contains multiple packet groups each containing image data of multiple regions of interest I, stored consecutively from the left.

[0124] 13, for example, in the captured image α, the region of interest ROI2 also exists on the same Y coordinate as the Y coordinate i1 (140) of the starting point coordinate of the region of interest ROI1. For this reason, as shown in FIG 15, in this example, in the line PL41 of the payload data, image data ROI2 (L41) of the 41st pixel row of the region of interest ROI2 and image data ROI1 (L1) of the 1st image row of the region of interest ROI1 are stored from the left in this order as image data of the pixel row corresponding to the Y coordinate 140 in the captured image α. Similarly, after the line PL41 of the payload data, in the lines PL42 (not shown) to PL200, image data of the two regions of interest ROI2 and ROI1 are stored from the left in this order.

[0125] 13, for example, in the captured image α, the regions of interest ROI2 and ROI1 are also present on the same Y coordinate as the Y coordinate i1 (300) of the starting point coordinate of the region of interest ROI0. For this reason, as shown in FIG. 15, in this example, in the line PL201 of the payload data, as image data of the pixel row corresponding to the Y coordinate 300 in the captured image α, image data ROI2 (L201) of the 201st pixel row of the region of interest ROI2, image data ROI0 (L1) of the 1st pixel row of the region of interest ROI0, and image data ROI1 (L161) of the 161st image row of the region of interest ROI1 are stored from the left in this order. Similarly, after the line PL201 of the payload data, in the lines PL202 (not shown) to PL600, image data of the three regions of interest ROI2, ROI0, and ROI1 are stored from the left in this order.

[0126] Moreover, the line PL600 of the payload data includes image data ROI2(L600) of the 600th pixel row, which is the last row, of the region of interest ROI2, as image data of the Y coordinate l2 of the end point coordinates of the region of interest ROI2 (Y coordinate=699 in the captured image α). Thereafter, the lines PL601 to PL640 of the payload data store image data of the two regions of interest ROI0 and ROI1, left-justified in this order. The line PL640 of the payload data includes image data ROI1(L600) of the 600th pixel row, which is the last row, of the region of interest ROI1, as image data of the Y coordinate l1 of the end point coordinates of the region of interest ROI1 (Y coordinate=739 in the captured image α). Thereafter, the lines PL641 to PL800 of the payload data store image data of one region of interest ROI0, left-justified in this order. Payload data line PL800 includes image data ROI0(L600) of the 600th pixel row, which is the last row of attention region ROI0, as image data of Y coordinate l0 (Y coordinate=899 in captured image α) of the end point coordinate of attention region ROI0. In this way, payload data for 800 lines including image data of attention regions ROI0 to 2 in this example is configured.

[0127] In this way, payload data storing image data of a predetermined number (three in this example) of regions of interest (ROIs) cut out from a predetermined captured image (captured image α in this example) is transmitted from the video transmitting device to the video receiving device together with embedded data including ROI information corresponding to the cut-out predetermined number of ROIs.

[0128] 3-2. Restoration process in video receiving device In this embodiment, when the video receiving device receives transmission data including image data of the attention area, the video receiving device performs restoration processing of the image data of the attention area included in the received transmission data. Between the video transmitting device and the video receiving device, the transmission data including the image data is communicated in units of image data frames. For example, data in each of the header and packet areas of the image data frame is transmitted to the video receiving device in units of transactions. In the video receiving device, various processes related to the image data included in the transmission data are performed by receiving a packet (SOF: Start Of Frame) indicating the start of the image data frame, and the restoration processing is one example of such processes.

[0129] In the restoration process in this embodiment, ROI information corresponding to the region of interest is extracted from the received transmission data, and ROI restoration information used in the restoration process is generated based on the extracted ROI information. The image data of the region of interest is decoded from the compressed state in the video transmission device, restored to a displayable state based on the ROI restoration information, and written to the frame memory.

[0130] By executing the restoration process, image data of the attention areas is restored to a state (displayable state) that can be displayed on a predetermined display device based on ROI information corresponding to a predetermined number of attention areas extracted from the embedded data. This makes it possible to efficiently display image data of the attention areas cut out at any position and image size in the captured image on the display device. For ease of understanding, the restoration process will be described below separately as an information extraction process and a writing process. The information extraction process is a process of generating ROI restoration information used in the restoration process from the extracted ROI information, and the writing process is a process of writing image data of the attention areas to a frame memory based on the ROI restoration information.

[0131] [Information Extraction Process] In the information extraction process, first, the embedded data in the transmission data is analyzed, and ROI information corresponding to a predetermined number of regions of interest (ROIs) cut out from the captured image is extracted from the embedded data. In the information extraction process, position information and size information of the regions of interest are extracted as ROI information from the embedded data.

[0132] In this embodiment, the position information is information indicating the position of the attention area in the captured image, and corresponds to, for example, the starting coordinates (X-coordinate and Y-coordinate) indicating the upper left end of the attention area. In addition, in this embodiment, the size information is information indicating the size (number of pixels) of the attention area in the captured image, and corresponds to, for example, the X-axis length and the Y-axis length of the attention area. Therefore, in the information extraction process in this embodiment, the starting coordinates of the attention area are extracted as the position information in the ROI information. In addition, the X-axis length and the Y-axis length of the attention area are extracted as the size information in the ROI information.

[0133] In this example, three regions of interest ROI0 to 2 are cut out from the captured image α as shown in Fig. 13. Therefore, in this example, in the information extraction process, the start coordinates, the X-axis length, and the Y-axis length (see Fig. 14) of each of the regions of interest ROI0 to 2 are extracted as ROI information from the embedded data.

[0134] (ROI restoration information) In the information extraction process, when ROI information (position information, size information) is extracted, ROI restoration information used for restoring image data of the region of interest is generated based on the extracted ROI information. Here, the ROI restoration information will be described in detail with reference to FIG. 16. FIG. 16 is a diagram for explaining the first coordinate data and the second coordinate data constituting the ROI restoration information. In the information extraction process, when generating the ROI restoration information, the first coordinate data and the second coordinate data shown in FIG. 16 are generated for each region of interest based on the extracted ROI information (position information and size information). For example, in this example, three regions of interest ROI0 to 2 are the targets of the restoration process. Therefore, when generating the ROI restoration information in this example, the first coordinate data and the second coordinate data corresponding to each of the three regions of interest ROI0 to 2 are generated.

[0135] The first coordinate data is composed of ROI identification data for identifying a region of interest and relative coordinate data indicating a relative position of the region of interest in the captured image. The first coordinate data is data in which the ROI identification data and the relative coordinate data are linked. The ROI identification information is information that enables each of a plurality of regions of interest in the transmission data that is the target of the restoration process to be identified, and is, for example, an arbitrary numerical value or a character string.

[0136] As shown in Fig. 16, the relative coordinate data is composed of a start coordinate indicating the upper left corner and an end coordinate indicating the lower right corner of the region of interest in the captured image. Specifically, the relative coordinate data included in the first coordinate data is composed of four numerical values ​​indicating the X coordinate of the start coordinate (start X coordinate), the Y coordinate of the start coordinate (start Y coordinate), the X coordinate of the end coordinate (end X coordinate), and the Y coordinate of the end coordinate (end Y coordinate). The relative coordinate data in the first coordinate data is, for example, a numerical value array in which the numerical values ​​indicating the start coordinate (X, Y) and the end coordinate (X, Y) are arranged with a comma separating them in the order shown in the upper part of Fig. 16.

[0137] As described above, in this embodiment, the ROI information sent from the video transmission device includes information on the start point coordinate as position information, but does not include information on the end point coordinate. Therefore, in the information extraction process, when generating the first coordinate data, the end point coordinate is calculated from the start point coordinate included in the position information and the X-axis length and Y-axis length included in the size information.

[0138] The second coordinate data is composed of ROI identification data and relative coordinate data, similar to the first coordinate data. However, as shown in Fig. 16, the relative coordinate data of the second coordinate data is composed of a start point Y coordinate and an end point Y coordinate, and differs from the first coordinate data in that it does not include a start point X coordinate and an end point X coordinate. The relative coordinate data in the second coordinate data is, for example, a numerical array in which numerical values ​​indicating the start point Y coordinate and the end point Y coordinate are arranged with a comma separating them in the order shown in the lower part of Fig. 16.

[0139] In the information extraction process of this embodiment, first and second coordinate data corresponding to a region of interest (ROI) to be restored are generated, and then X-coordinate ordered data is generated by rearranging the first coordinate data of each region of interest in ascending order of the starting X-coordinate. Also, Y-coordinate ordered data is generated by rearranging the second coordinate data in ascending order of the starting Y-coordinate. In this embodiment, the X-coordinate ordered data and the Y-coordinate ordered data correspond to ROI restoration information.

[0140] In this way, the ROI restoration information is information generated by rearranging the position information and size information in the ROI information. More specifically, the ROI restoration information is information generated by rearranging data (first coordinate data, second coordinate data) generated from the position information and size information. In the information extraction process in this embodiment, X-coordinate order data and Y-coordinate order data are generated as the ROI restoration information as described above. The generated X-coordinate order data and Y-coordinate order data are temporarily stored in a predetermined storage area (e.g., a register, etc.) of the video receiving device, and are used in the subsequent restoration process (e.g., a write process).

[0141] Here, a specific example of the X-coordinate ordered data and the Y-coordinate ordered data will be described with reference to Fig. 17. Fig. 17 is a diagram showing an example of the X-coordinate ordered data and the Y-coordinate ordered data corresponding to the regions of interest ROI0 to 2 that are targets of the restoration process in this example.

[0142] In the information extraction process in this example, when generating the first coordinate data and the second coordinate data constituting the X-coordinate ordered data and the Y-coordinate ordered data, the end point coordinates of the regions of interest ROI0 to 2 are calculated. The end point X coordinate of the region of interest is calculated as "start point X coordinate + X-axis length - 1". As shown in FIG. 14, the value of the start point X coordinate i0 of the region of interest ROI0 is "520", and the length of the X-axis direction is "400". Therefore, the value of the end point X coordinate k0 is calculated as "919" = "start point X coordinate i0 (520) + X-axis length (400) - 1". Similarly, the value of the end point X coordinate k1 of the region of interest ROI1 is "1439", and the value of the end point X coordinate k2 of the region of interest ROI2 is "499".

[0143] Also, the end point Y coordinate of the region of interest is calculated by "start point Y coordinate + Y axis length - 1". As shown in FIG. 14, the value of the start point Y coordinate j0 of the region of interest ROI0 is "300", and the length in the X axis direction is "600". Therefore, the value of the end point Y coordinate l0 is calculated as "899" = "start point Y coordinate i0 (300) + Y axis length (600) - 1". Similarly, the value of the end point Y coordinate l1 of the region of interest ROI1 is calculated as "739", and the value of the end point Y coordinate l2 of the region of interest ROI2 is calculated as "699". In this manner, the end point coordinates of each region of interest are calculated.

[0144] In the information extraction process in this example, when the end point coordinates are calculated, first coordinate data and second coordinate data of the regions of interest ROI0 to 2 are generated using the start point coordinate information included in the ROI information and the calculated end point coordinates.

[0145] In this example, the first coordinate data of the region of interest ROI0 is represented by "R0 (520, 300, 919, 899)". Here, "R0" is ROI identification information indicating the region of interest ROI0. Similarly, the first coordinate data of the region of interest ROI1 is represented by "R1 (1040, 140, 1439, 739)", and the first coordinate data of the region of interest ROI2 is represented by "R2 (100, 100, 499, 699)". When the start point X coordinates of the regions of interest ROI0 to 2 are arranged in ascending order, the order is "100" indicating the start point X coordinate i2 of the region of interest ROI2, "520" indicating the start point X coordinate i0 of the region of interest ROI0, and "1040" indicating the start point X coordinate i1 of the region of interest ROI1. Therefore, by rearranging the first coordinate data of the regions of interest ROI0 to 2 in ascending order of the starting X coordinate (in the order of the regions of interest ROI2, ROI0, ROI1), X-coordinate ordered data corresponding to the regions of interest ROI0 to 2 shown on the left side of FIG. 17 is generated.

[0146] In this example, the second coordinate data of the region of interest ROI0 is represented by "R0 (300, 899)". Similarly, the second coordinate data of the region of interest ROI1 is represented by "R1 (140, 739)", and the second coordinate data of the region of interest ROI2 is represented by "R2 (100, 699)". When the start point Y coordinates of the regions of interest ROI0 to 2 are arranged in ascending order, the order is "100" indicating the start point Y coordinate l2 of the region of interest ROI2, "140" indicating the start point Y coordinate l1 of the region of interest ROI1, and "899" indicating the start point Y coordinate l0 of the region of interest ROI1. Therefore, by rearranging the second coordinate data of the regions of interest ROI0 to 2 in ascending order of the start point Y coordinate (the order of the regions of interest ROI2, ROI1, ROI0), the Y coordinate order data corresponding to the regions of interest ROI0 to 2 shown on the right side of FIG. 17 is generated. In this example, the X-coordinate ordered data and the Y-coordinate ordered data corresponding to the generated regions of interest ROI0 to 2 are stored in the register while maintaining the order shown in FIG.

[0147] In Fig. 17, for ease of understanding, the order of the X-coordinate ordered data and the Y-coordinate ordered data is indicated by the numbers 1 to 3 at the left end. Also, in Fig. 17, for ease of understanding, the X-coordinate ordered data is displayed in a table format in which the first coordinate data of the regions of interest ROI0 to 2 are stored in ascending order of the start point X-coordinate on a row-by-row basis, and the Y-coordinate ordered data is displayed in a table format in which the second coordinate data of the regions of interest ROI0 to 2 are stored in ascending order of the start point Y-coordinate on a row-by-row basis, but the storage format of the X-coordinate ordered data and the Y-coordinate ordered data is not limited to this. In this embodiment, it is sufficient that the X-coordinate ordered data and the Y-coordinate ordered data are stored so that the order of a predetermined number of regions of interest is maintained.

[0148] For example, the X-coordinate order data may store the first coordinate data of each attention area together with a numerical value indicating the order when the data are rearranged in ascending order of the starting X-coordinate. Similarly, the Y-coordinate order data may store the second coordinate data of each attention area together with a numerical value indicating the order when the data are rearranged in ascending order of the starting Y-coordinate. Also, the X-coordinate order data may be stored as an array in which the first coordinate data of each attention area is separated by a predetermined delimiter (e.g., a comma). Similarly, the Y-coordinate order data may be stored as an array in which the second coordinate data of each attention area is separated by a predetermined delimiter (e.g., a comma).

[0149] In the information extraction process of this example, by generating X-coordinate order data and Y-coordinate order data corresponding to the regions of interest ROI0-2, it becomes possible to determine the relative positions of the regions of interest ROI0-2 to be restored in the captured image α (see FIG. 13). This makes it possible to perform processing using the relative positions of the regions of interest ROI0-2 in the restoration process (write process) following the information extraction process.

[0150] In this way, the X-coordinate ordered data is data in which the position information and size information of a predetermined number of regions of interest (ROIs) cut out from a captured image are rearranged in ascending order of the X-coordinate value of the starting point coordinate (starting point X-coordinate) corresponding to the position information. More specifically, the X-coordinate ordered data is data in which the relative coordinate information generated from the position information and size information of the regions of interest are rearranged in ascending order of the starting point X-coordinate. Moreover, the Y-coordinate ordered data is data in which the position information and size information of a predetermined number of regions of interest cut out from a captured image are rearranged in ascending order of the Y-coordinate value of the starting point coordinate corresponding to the position information. More specifically, the Y-coordinate ordered data is data in which the Y-coordinate in the relative coordinate information generated from the position information and size information of the regions of interest are rearranged in ascending order of the starting point Y-coordinate.

[0151] In the information extraction process of the restoration process, the ROI restoration information is generated, so that the relative position of the region of interest in the captured image can be determined. This allows smooth processing in the subsequent processes of the restoration process (e.g., the writing process) using the relative position. As will be described in detail later, the stored X-coordinate order data and Y-coordinate order data are updated every time the video receiving device receives transmission data and starts a new information extraction process.

[0152] [Writing Process] Next, the writing process in the restoration process will be described. The writing process is a process of writing image data of the region of interest into a memory device in the video receiving device based on the ROI restoration information generated in the information extraction process. In this embodiment, a frame memory is assumed as the memory device into which the image data of the ROI is written. The frame memory is a memory IC capable of holding image data to be displayed on a display device (e.g., a display) on a screen-by-screen basis.

[0153] In this embodiment, as described above, a frame for communicating transmission data is started by receiving an SOF, and when the transmission data is received in the video transmitting device, the embedded data is input first, and then each line of payload data is input in order from the beginning. The above-mentioned information extraction process is executed based on the input of the embedded data, but the writing process in the restoration process is executed based on the input of each line of payload data. In the writing process in this embodiment, image data of the region of interest is written to the frame memory in payload data units based on the ROI restoration information. In the writing process, ROI pixel row data of each line of payload data is written to the frame memory.

[0154] In this embodiment, in the write process of the restoration process, image data of the region of interest in the payload data, i.e., ROI pixel row data, is divided for each region of interest based on ROI restoration information. That is, the ROI pixel row data is divided into a packet group for each region of interest, i.e., image data of the region of interest (ROI), based on the ROI restoration information. In the write process, an address indicating a write destination of the image data of the region of interest (ROI) in the frame memory is generated based on the ROI restoration information. The image data of each region of interest divided from the ROI pixel row data is written to the corresponding address in the frame memory.

[0155] In this embodiment, as will be described in detail later, a restoration position in the frame memory corresponding to the relative position of each region of interest in the captured image is specified as a writing destination of image data of the region of interest based on the ROI restoration information, and an address indicating the restoration position is generated. As a result, the relative position of the region of interest in the captured image is restored in the frame memory. Therefore, the image data of the region of interest can be displayed on the display screen at a display position corresponding to the relative position in the captured image.

[0156] (ROI restoration information analysis) In the write process in this embodiment, the ROI restoration information (X-coordinate order data, Y-coordinate order data) is analyzed, and information corresponding to the ROI pixel row data in each line of the payload data is extracted based on the ROI restoration information to generate information used for writing (write information). Based on the write information generated from the ROI restoration information, the ROI pixel row data is divided, and addresses indicating the write destinations in the frame memory of the divided ROI pixel row data are generated. In this embodiment, for example, the write Y coordinate, the write start X coordinate, and the write size are generated as the write information. In the write process of the restoration process, the X-coordinate order data and the Y-coordinate order data are extracted for each input of each line of the payload data to generate the write information. The write information (write Y coordinate, write start X coordinate, and write size) will be specifically described below.

[0157] (Y coordinate to write) The write Y coordinate is the Y coordinate of the ROI pixel row data in the captured image. The write Y coordinate is used to determine the position in the Y axis direction in the frame memory when writing the ROI pixel row data. The ROI pixel row data stored in each line of the payload data is composed of image data of the region of interest at the same Y coordinate in the captured image. Therefore, the write Y coordinate is a common value for each line of the payload data. The write Y coordinate is generated from the Y coordinate order data of the ROI restoration information. For example, when each line of the payload data is input, second coordinate data is extracted from the Y coordinate order data in the storage order (ascending order of the starting point X coordinate). Using the extracted second coordinate data, a write Y coordinate corresponding to each line of the input payload data is generated.

[0158] In this example, when line PL1 of payload data (see FIG. 15) is input, second coordinate data relating to the first region of interest ROI2 of the Y coordinate ordered data (see the right side of FIG. 17) is extracted, and a write Y coordinate indicating the start Y coordinate (100) of the region of interest ROI2 is generated. That is, when the top line (first line) of payload data is input, the first second coordinate data of the Y coordinate ordered data is extracted, and the start Y coordinate in the second coordinate data, i.e., the start Y coordinate with the smallest value in the Y coordinate ordered data, is determined to be the write Y coordinate.

[0159] After the first line is input, the value of the write Y coordinate is updated by incrementing it by 1 each time a new line of payload data is input. Also, when the updated value of the write Y coordinate exceeds the value of the end point Y coordinate, the next second coordinate data is extracted from the Y coordinate ordered data. In this example, when line PL601 of the 601st row of the payload data is input, the updated value of the write Y coordinate becomes "700", which exceeds the value of the end point Y coordinate of the region of interest ROI2, "699". Therefore, the second coordinate data related to the second region of interest ROI1 is extracted from the Y coordinate ordered data (see the right side of FIG. 17).

[0160] When new second coordinate data is extracted, it is determined whether the updated value of the write Y coordinate is included within the range from the start Y coordinate to the end Y coordinate of the newly extracted second coordinate data. If the updated value of the write Y coordinate is within the range, the updated value of the write Y coordinate is used as the write Y coordinate corresponding to the input line. In this example, it is determined that the updated value of the write Y coordinate "700" is within the range from the start Y coordinate value "140" to the end Y coordinate value "739" of the region of interest ROI1. Therefore, the updated value of the write Y coordinate "700" is used as the write Y coordinate of the ROI pixel row data of the line PL601 of the payload data.

[0161] On the other hand, if the updated value of the write Y coordinate is not included in the range, the updated value of the write Y coordinate does not correspond to the Y coordinate of the ROI pixel row data in the input line. Therefore, the write Y coordinate is generated based on the newly extracted second coordinate data. For example, if the updated value of the write Y coordinate is smaller than the value of the start point Y coordinate of the newly extracted second coordinate data, the write Y coordinate may be generated with the value of the start point Y coordinate of the second coordinate data. Also, if the updated value of the write Y coordinate is larger than the value of the end point Y coordinate of the newly extracted second coordinate data, the next second coordinate data may be extracted from the Y coordinate order data to generate the write Y coordinate. This allows the Y coordinate position in the captured image of each attention area to be reflected in the write Y coordinate.

[0162] (Write start X coordinate and write size) Next, the generation of the write start X coordinate and the write size will be described. In contrast to the write Y coordinate generated for each ROI pixel row, the write start X coordinate and the write size are generated for each region of interest. The write start X coordinate indicates each packet group constituting the ROI pixel row data, that is, the X coordinate of the image data of each region of interest in the captured image. The write size indicates the image size (here, the length in the X-axis direction) of the image data of each region of interest. When writing the ROI pixel row data to the frame memory, the ROI pixel row data is divided for each region of interest based on the write start X coordinate and the write size. For this reason, the write size is also called the "division size."

[0163] The write start X coordinate is used to calculate the write start position (top address) in the X-axis direction in the frame memory corresponding to each region of interest when writing image data of each region of interest. The write size is used to calculate consecutive addresses in the frame memory for storing image data of the region of interest divided from the ROI pixel row data.

[0164] The write start X coordinate and the write size are generated from the X coordinate order data of the ROI restoration information. For example, when each line of payload data is input, the first coordinate data is extracted from the X coordinate order data in the storage order (ascending order of the start point X coordinate). For the extracted first coordinate data, it is determined whether the value of the current write Y coordinate is included within the range from the start point Y coordinate to the end point Y coordinate of the extracted first coordinate data. This determines whether the image data of each attention area is included in the ROI pixel row data of each line. Here, the attention area associated with the image data included in the ROI pixel row data is called the write target ROI. In this embodiment, when a specific line of payload data is input, the value of the write Y coordinate is first generated (updated), and then the write start X coordinate and the write size are generated.

[0165] In this example, when the first line PL1 (see FIG. 15) of the payload data is input, first, the first coordinate data related to the first region of interest ROI2 of the X-coordinate ordered data (see the right side of FIG. 17) is extracted. At this time, as described above, the value of the write Y coordinate is "100", and is included in the range from the value of the start point Y coordinate "100" of the region of interest ROI2 to the value of the end point Y coordinate "699". Therefore, the ROI pixel row data of the line PL1 includes image data of the region of interest ROI2, and the region of interest ROI2 is determined to be the ROI to be written. On the other hand, the value of the write Y coordinate "100" is not included in the range from the value of the start point Y coordinate "140" of the first coordinate data related to the second region of interest ROI1 of the X-coordinate ordered data to the value of the end point Y coordinate "739". Similarly, the value of the write Y coordinate "100" is not included in the range from the value of the start point Y coordinate "300" of the first coordinate data related to the third region of interest ROI0 of the X-coordinate ordered data to the value of the end point Y coordinate "899". Therefore, the regions of interest ROI0 and ROI1 are determined not to be ROIs to be written.

[0166] In this embodiment, a write start X coordinate and a write size are generated for each region of interest (ROI) determined to be a write target ROI. Specifically, the start X coordinate of the write target ROI is generated as the write start X coordinate. In this example, the write start X coordinate of the region of interest ROI2 is generated as the start X coordinate value "100". The write size is generated by adding 1 to the difference between the start X coordinate value and the end X coordinate value in the first coordinate data of the write target ROI. In this example, the write size of the region of interest ROI2 is "400 pixels (=end X coordinate (499)-start X coordinate (100)+1)".

[0167] If it is determined that the region of interest ROI0 is the ROI to be written in, the write start X coordinate of the region of interest ROI0 is generated as the start X coordinate value "300", and the write size of the region of interest ROI0 is "400 pixels (= end X coordinate (919) - start X coordinate (520) + 1)". If it is determined that the region of interest ROI1 is the ROI to be written in, the write start X coordinate of the region of interest ROI1 is generated as the start X coordinate value "1040", and the write size of the region of interest ROI1 is "400 pixels (= end X coordinate (1439) - start X coordinate (1040) + 1)".

[0168] In this example, when payload data is input from line PL1 to line PL40, the write target ROI is only the region of interest ROI2. Therefore, the write start X coordinate and write size of the region of interest ROI2 are generated. Furthermore, when payload data is input from line PL41 to line PL200, the write target ROI is the regions of interest ROI2 and ROI1. Therefore, the write start X coordinate and write size of each of the regions of interest ROI2 and ROI1 are generated. Furthermore, when payload data is input from line PL201 to line PL600, the write target ROI is the regions of interest ROI2, ROI0, and ROI1. Therefore, the write start X coordinate and write size of each of the regions of interest ROI2, ROI0, and ROI1 are generated.

[0169] Moreover, after the input of line PL601 of the payload data, the region of interest ROI2 no longer corresponds to the ROI to be written. Therefore, the write start X coordinate and write size of each of the regions of interest ROI0 and ROI1 are generated. Moreover, after the input of line PL641 of the payload data, the region of interest ROI1 no longer corresponds to the ROI to be written. Therefore, only the write start X coordinate and write size of the region of interest ROI0 are generated. In this way, in this embodiment, the write size of the image data of the region of interest (ROI) to be written to the frame memory is calculated based on the position information in the ROI restoration information (X coordinate order data in this example).

[0170] (Frame memory address generation) In the write process in the restoration process of this embodiment, when write information (write Y coordinate, write start X coordinate, and write size) is generated from the ROI restoration information, an address for writing image data of the region of interest (write target ROI) included in the ROI pixel row data in the frame memory is specified based on the write information. In this embodiment, as described above, an address corresponding to a restoration position indicating the relative position of the region of interest in the captured image is generated in the frame memory. In addition, the address of the image data of the write target ROI is generated in the order of storage of the X coordinate order data (in ascending order of the starting point X coordinate).

[0171] For example, the Y-axis position in the frame memory among the restoration positions of each region of interest (here, the write target ROI) is specified based on the write Y coordinate that is common to each ROI pixel row. Also, the start position in the X-axis direction in the frame memory among the restoration positions of each write target ROI is specified based on the write start X coordinate for each write target ROI. This specifies the write start position of the image data of the write target ROI in the frame memory.

[0172] In this example, when the first line PL41 (see FIG. 15) of the payload data is input, the write Y coordinate "140" for the ROI pixel row data, the write start X coordinate "100" for the region of interest ROI2, and the write start X coordinate "140" for the region of interest ROI1 are generated. In this case, the position corresponding to the pixel coordinate "100, 140 (X coordinate, Y coordinate)" in the captured image α (see FIG. 13) is determined as the write start position in the frame memory of the image data of the region of interest ROI2 included in the ROI pixel row data in PL41. Also, the position corresponding to the pixel coordinate "520, 140 (X coordinate, Y coordinate)" in the captured image α is determined as the write start position in the frame memory of the image data of the region of interest ROI1 included in the ROI pixel row data.

[0173] In this way, the write start position is determined for each write target ROI in the ROI pixel row data, and the top address in the frame memory indicating the write start position is calculated. In this embodiment, the write area of ​​the image data in the frame memory can be applied to pixel coordinates equivalent to the captured image, for example. In this case, for example, by identifying the coordinates of the write start position, the address indicating the coordinates in the frame memory can be calculated. Note that the specific calculation method (calculation formula) of the address may be appropriately designed according to the specifications of the frame memory, etc.

[0174] Furthermore, using the start address of each region of interest and the write size generated for each region of interest, consecutive addresses for the number of pixels indicated by the write size are calculated. For example, the value of the start address is incremented by 1 for the number of pixels indicated by the write size. This calculates consecutive addresses for the number of pixels indicated by the write size. By writing each pixel of the image data of the write target ROI to these consecutive addresses, a series of image data of the write target ROI can be written to the frame memory. In the frame memory, the area from the write start position to the write size corresponds to the restoration position in pixel row units of the write target ROI.

[0175] In this example, when line PL41 (see FIG. 15) of the 41st row of the payload data is input, the start address of the region of interest ROI2 is added repeatedly to calculate an address of the write size "400 pixels". As a result, consecutive addresses of 400 pixels are generated from the start address of the region of interest ROI2 in the frame memory. Similarly, consecutive addresses of 400 pixels are generated from the start address of the region of interest ROI1. As a result, image data of 400 pixels each of the regions of interest ROI2 and ROI1 is written into the frame memory.

[0176] (Dividing image data) In the write process in this embodiment, image data (ROI pixel row data) of a region of interest (ROI) in the payload data is divided for each region of interest based on the ROI restoration information, transferred, and written to the frame memory. Specifically, based on the write information (write Y coordinate, write start X coordinate, and write size) generated from the ROI restoration information, the ROI pixel row data in each line of the payload data is divided for each region of interest (write target ROI). In the write process in the restoration processing in this embodiment, the ROI pixel row data is divided for each region of interest based on the write size of the write information.

[0177] For example, image data of the write target ROI is divided from the ROI pixel row data in the order of storage of the X-coordinate order data (starting from the smallest starting point X-coordinate). That is, the ROI pixel row data is divided into image data for each write target ROI in order from the payload header side. The divided image data for each write target ROI is separated from the ROI pixel row data for each write target ROI, and the write target image data is generated. The write target image data is transferred to the frame memory and written to an address generated based on the write information. Therefore, the ROI pixel row data is written to the frame memory with a number of writes corresponding to the number of divisions for each write target ROI.

[0178] In this example, when the 41st line PL41 (see FIG. 15) of the payload data is input, the ROI pixel row data is divided based on the write size of each of the regions of interest ROI2 and ROI1, which are the ROIs to be written. The pixel row data of the line PL41 is divided into two, image data of the region of interest ROI2 (ROI2(L41)) and image data of ROI1 (ROI1(L1)), starting from the head of the ROI pixel row data, that is, the payload header PH side of the line PL41. Therefore, the number of times that the ROI pixel row data of the line PL41 is written to the frame memory is two. Specifically, in the 41st line PL41 of the payload data, the division size (write size) of the region of interest ROI2 is 400 pixels, and the division size of the region of interest ROI1 is 400 pixels. Therefore, the image data of 400 pixels from the head of the ROI pixel row data is divided to generate the image data to be written of the region of interest ROI2. Further, image data to be written in the region of interest ROI1 is generated from image data of the remaining 400 pixels in the ROI pixel row data.

[0179] The image data to be written generated for each region of interest (ROI to be written) is transferred to the frame memory in the order in which it was divided from the ROI pixel row data (in ascending order of the write start X coordinate), for example, and written to the address indicating the restoration position corresponding to each region of interest (ROI). As a result, the image data to be written of the ROI to be written is written to the restoration position in the frame memory corresponding to the ROI to be written. Note that when there is one ROI to be written (for example, when inputting line PL1 of the first line of payload data), the pixel row data can be transferred to the frame memory without being divided, but it may be divided appropriately in consideration of the bus specifications and the number of pixels in the pixel row data.

[0180] The write process in the restoration process of this embodiment has been described above. As described above, every time each line of payload data is input, the address of the write target ROI and the division of the ROI pixel row data are performed, and the ROI pixel row data of all lines of the payload data are written into the frame memory. As a result, each of the attention areas cut out from the captured image is written to a restored position in the frame memory. Therefore, the image data of the attention area is restored to a state in which it can be displayed on a predetermined display device in the video receiving device. In this example, the attention areas ROI0 to 2 are written into the restored positions corresponding to their relative positions in the captured image α (see FIG. 13) in the frame memory. That is, the attention areas ROI0 to 2 are restored to a state in which they can be displayed on the screen of the predetermined display device at display positions corresponding to their relative positions in the captured image α (see FIG. 13).

[0181] In this way, in this embodiment, after going through the information extraction process and writing process in the restoration process, the image data of the area of ​​interest is restored to a displayable state and written to the frame memory, thereby enabling the image data to be efficiently displayed on a display device such as a monitor.

[0182] 4. Video transmission system of the present disclosure 4-1. Overview of video transmission system Next, a video transmission system and a video receiving device according to this embodiment will be described with reference to Fig. 18 and Fig. 19. First, a schematic configuration of the video transmission system according to this embodiment will be described with reference to Fig. 18. Fig. 18 is a block diagram showing a schematic configuration of a video transmission system (an example of a transmission system) 2. Fig. 19 is a block diagram showing an example of the configuration of a video receiving device 200a according to this embodiment.

[0183] 18, the video transmission system 2 includes a video transmitting device (an example of a transmitting device) 100 and a video receiving device (an example of a receiving device) 200a. Note that components having the same actions and functions as those in the video transmission system 1 according to the above-mentioned base technologies 1 and 2 (see FIG. 1) are denoted by the same reference numerals and will not be described.

[0184] In the video transmission system 2, the video transmitting device 100 generates transmission data including embedded data and payload data in which image data of a region of interest and data of a normal image are stored. The transmission data generated in the video transmitting device 100 is sent to the video receiving device 200a via the data lane DL. When the video receiving device 200a receives the transmission data, it performs a predetermined process on the transmission data. When the video receiving device 200a according to this embodiment receives the transmission data including image data of a region of interest, it performs a restoration process on the image data of the region of interest.

[0185] In this embodiment, the video receiving device 200a is connected to the display device 300. The display device 300 is, for example, a display device having a screen capable of displaying images. The video receiving device 200a outputs image data (normal image data, image data of a region of interest (ROI)) included in the transmission data to the display device 300. As a result, for example, image data of the region of interest cut out from a captured image by the video transmitting device 100 and sent as the transmission data is restored by the video receiving device 200a and then displayed on the screen of the display device 300.

[0186] Next, a video receiving device 200a according to this embodiment will be described. The video receiving device 200a is different from the video receiving device 200 (see FIG. 8) according to the prerequisite technologies 1 and 2 in that the video receiving device 200a is configured to be able to execute a restoration process. In the video receiving device 200a, components that perform the same actions and functions as those of the video receiving device 200 according to the prerequisite technologies 1 and 2 are given the same reference numerals and descriptions thereof will be omitted as appropriate.

[0187] As shown in Fig. 19, the video receiving device 200a includes, for example, a receiving unit 210, an information processing unit (an example of a processing unit) 230, and a frame memory 250. Although details will be described later, in this embodiment, the information processing unit 230 is composed of a first information processing unit 230a and a second information processing unit 230b. The information processing unit 230 (first information processing unit 230a, second information processing unit 230b) and the frame memory 250 are connected to each other so as to be able to transfer image data via a bus 240. In addition, the second information processing unit 230b and the frame memory 250 are connected to each other so as to be able to transfer image data via the bus 240 under the control of the DMA controller 20.

[0188] The receiving unit 210 in the video receiving device 200a receives a transmission signal (transmission data 147A in this example) including image data of a predetermined number of regions of interest (ROIs) cut out from a predetermined captured image in payload data and including ROI information corresponding to the predetermined number of regions of interest in embedded data. As with the video transmitting device 100 in the base technologies 1 and 2, the video transmitting device 100 (see FIG. 18) constituting the transmission system 2 according to this embodiment transmits image data of each region of interest (ROI) in a captured image 111 as payload data of a long packet and transmits ROI information 120B for the region of interest as embedded data. This allows the device (video receiving device 200) that receives the transmission data 147A transmitted from the video transmitting device 100 to easily extract image data (ROI image 112) of each region of interest ROI from the transmission data 147A. The receiving unit 210 is a circuit that performs a predetermined process on the received transmission data 147A to generate various data (214A, 215A, 215B) and outputs the data to the information processing unit 220. In this embodiment, when the receiving unit 210 receives a packet (SOF) indicating that communication of an image data frame with the video transmitting device 100 has started, the receiving unit 210 outputs a predetermined signal indicating reception of the SOF to the first information processing unit 230a.

[0189] As in the video receiving device 200 in the above-mentioned base technologies 1 and 2, in the video receiving device 200a, the receiving unit 210 includes, for example, a header separation unit 211, a header interpretation unit 212, a payload separation unit 213, an EBD interpretation unit 214, and an ROI data separation unit 215. Of these, the header separation unit 211 and the header interpretation unit 212 have the same configuration as in the video receiving device 200, and therefore a description thereof will be omitted.

[0190] The EBD interpretation unit 214 determines whether the image data included in the payload data of the transmission data 147A is image data of the ROI (compressed image data 120A of the image data 116 of the region of interest in the prerequisite technologies 1 and 2) or data of a normal image (compressed image data 130A) from the data type included in the embedded data 213A input from the payload separation unit 213. The EBD interpretation unit 214 also outputs the data type determination result to the ROI data separation unit 215. The EBD interpretation unit 214 also outputs embedded data including ROI information (indicating that the image data of the region of interest is included in the payload data) as EBD data 214A to the first information processing unit 230a (specifically, the first information extraction unit 231).

[0191] When the ROI data separation unit 215 determines that the image data included in the payload data of the transmission data 147A is image data of the region of interest (compressed image data 120A of the image data 116) based on the data type determination result by the EBD interpretation unit 214, it outputs each line of the payload data as payload data 215A to the first information processing unit 230a (specifically, the ROI decoding unit 232) line by line. The payload data 215A includes one line of pixel data (ROI pixel row data) from the compressed image data 147B.

[0192] In this embodiment, for example, each line of payload data is transmitted to video receiving device 200a in transaction units constituting an image data frame related to transmission data 147A sent from video transmitting device 100. When the header of each line of payload data of a long packet is detected in header interpretation section 212 of receiving section 210, each line of payload data is separated from packet area R2 in payload separation section 213, and payload data 215A including the payload header PH, ROI pixel row data and payload footer PF of each line is generated in ROI data separation section 215.

[0193] 15 is transmitted from video transmitting device 100, payload data 215A generated by ROI data separation unit 215 includes pixel data for one line among lines PL1-800 of the payload data. At this time, ROI data separation unit 215 outputs the payload data of lines PL1-800 constituting the packet area of ​​the image data frame in order from line PL1 to ROI decoding unit 232 as payload data 215A. When ROI information 120B is included in the payload data of the long packet, payload data 215A may include ROI information 120B.

[0194] On the other hand, when the image data included in the payload data is compressed image data 130A of normal image data, the ROI data separation unit 215 outputs the payload data as payload data 215B to the first information processing unit 230a (specifically, the normal image decoding unit 236). In addition, the ROI data separation unit 215 outputs the determination result of the data type input from the EBD interpretation unit 214 (the determination result indicating that the image data is normal image data) to the first information processing unit 230a (specifically, the normal image address generation unit 235).

[0195] The first information processing unit 230a is a circuit that performs a restoration process to restore image data of the regions of interest (payload data 215A) to a state that can be displayed on the display device 300, based on ROI information corresponding to a predetermined number of regions of interest (ROIs) extracted from the embedded data. The first information processing unit 230a also performs a predetermined process (normal image processing) on ​​data of a normal image (payload data 215B).

[0196] The first information processing unit 230a has, as components related to the execution of the restoration process, a first information extraction unit 231, a ROI decoding unit 232, a ROI image address generation unit 233, and a ROI image generation unit 234. First, these components that perform the restoration process will be described.

[0197] The first information processing unit 230a extracts ROI information from the embedded data (EBD data 214A). The first information processing unit 230a executes the above-mentioned information extraction process and writing process in the restoration process, and executes a series of processes to restore image data of the region of interest (ROI) and write it to the frame memory 250.

[0198] The first information extraction unit 231 executes an information extraction process in the restoration process. Specifically, the first information extraction unit 231 extracts position information and size information of each of a predetermined number of regions of interest in the captured image as ROI information from the ROI information (ROI information 120B) included in the embedded data (EBD data 214A). In this example, the first information extraction unit 231 extracts the start point coordinates of each of the three regions of interest ROI0 to 2 in the captured image α as position information, and extracts the X-axis length and Y-axis length of each of the regions of interest ROI0 to 2 as size information.

[0199] The first information extraction unit 231 generates ROI restoration information (X-coordinate order data, Y-coordinate order data) used to restore image data of the region of interest based on the extracted ROI information. Specifically, the position information in the ROI information includes the X-coordinate and the Y-coordinate indicating the upper left end of the region of interest (ROI). The first information extraction unit 231 generates X-coordinate order data by rearranging the position information and size information of a predetermined number of regions of interest cut out from the captured image in ascending order of the value of the starting point X-coordinate. In this example, the first information extraction unit 231 generates first coordinate data (see FIG. 16) corresponding to each of the regions of interest ROI0 to 2, and generates X-coordinate order data (see the left side of FIG. 17) by rearranging the generated first coordinate data in ascending order of the value of the starting point X-coordinate (the order of the regions of interest ROI2, ROI0, ROI1).

[0200] The first information extraction unit 231 also generates Y coordinate ordered data by rearranging the position information and size information of a predetermined number of regions of interest cut out from the captured image in ascending order of the value of the starting point Y coordinate. In this example, the first information extraction unit 231 generates second coordinate data (see FIG. 16) corresponding to each of the regions of interest ROI0-2, and generates Y coordinate ordered data (see the right side of FIG. 17) by rearranging the generated second coordinate data in ascending order of the value of the starting point Y coordinate (the order of the regions of interest ROI2, ROI1, ROI0). After generating the X coordinate ordered data and the Y coordinate ordered data, the first information extraction unit 231 stores the generated X coordinate ordered data and Y coordinate ordered data in a register as ROI restoration information, for example.

[0201] The first information extraction unit 231 may generate final line determination information when generating ROI restoration information. The final line determination information is generated based on, for example, Y coordinate order data. The first information extraction unit 231 may compare, for example, end point Y coordinates in the second coordinate data in the Y coordinate order data, and store the Y coordinate with the largest value in a register as the final line Y coordinate. This allows the first information extraction unit 231 to use the final line determination information to determine whether writing of image data of the region of interest for all lines in the payload data has been completed.

[0202] Furthermore, after generating the restoration information in the information extraction process, the first information extraction unit 231 then analyzes the ROI restoration information to generate write information (write Y coordinate, write start X coordinate, write size) in the write process. The first information extraction unit 231 generates write information for the input payload data 215A every time the payload data 215A is input to the first information processing unit 230a (specifically, the ROI decoding unit 232). That is, the first information extraction unit 231 generates write information for each payload data unit. When the payload data 215A is input from the ROI data separation unit 215 based on the header of each line of the payload data being detected by the header interpretation unit 212, the ROI decoding unit 232 outputs a predetermined signal indicating the input to the first information extraction unit 231. This enables the first information extraction unit 231 to detect that each line of the payload data has been input.

[0203] In this example, every time payload data 215A for each line of lines PL1 to 800 of the payload data shown in FIG. 15 is input to the ROI decoding unit 232, the first information extraction unit 231 generates write information for the payload data 215A corresponding to each line.

[0204] The first information extraction unit 231 generates a write Y coordinate corresponding to the payload data 215A from the Y coordinate order data of the ROI restoration information as the write information. The method of generating the write Y coordinate is as described above. For example, when the first information extraction unit 231 detects an input of the payload data 215A corresponding to the line PL1 (see FIG. 15) of the payload data, it extracts the second coordinate data related to the first region of interest ROI2 from the Y coordinate order data (see the right side of FIG. 17) and determines the value "100" of the start point Y coordinate of the region of interest ROI2 as the write Y coordinate. After that, the first information extraction unit 231 updates (increments) the value of the write Y coordinate every time the payload data 215A is input. When the line PL601 of the 601st row of the payload data is input, the first information extraction unit 231 extracts the second coordinate data related to the second region of interest ROI1 from the Y coordinate order data and uses the updated value "700" of the write Y coordinate as the write Y coordinate.

[0205] In addition, the first information extraction unit 231 generates the write start X coordinate and the write size corresponding to the image data of each region of interest (write target ROI) included in the payload data 215A from the X coordinate order data in the ROI restoration information as the write information. The method of generating the write start X coordinate and the write size is as described above.

[0206] For example, suppose that the input of payload data 215A corresponding to line PL1 of the payload data is detected and the first information extraction unit 231 generates a write Y coordinate (Y=100). In this case, it is included in the range from the start point Y coordinate value "100" to the end point Y coordinate value "699" in the first coordinate data of the first region of interest ROI2 in the X-coordinate order data. Therefore, the first information extraction unit 231 determines that the region of interest ROI2 is the ROI to be written, and sets the start point X coordinate (X=100) of the region of interest ROI2 as the write start X coordinate of the image data of the region of interest ROI2. In addition, the first information extraction unit 231 calculates "400 pixels (=end point X coordinate (499)-start point X coordinate (100)+1)" as the write size of the region of interest ROI2.

[0207] When the first information extraction unit 231 determines the write target ROI in the payload data 215A based on the X-coordinate order data, it generates write information (write Y coordinate, write start X coordinate, and write size) for each write target ROI, and outputs the generated write information to the ROI image address generation unit 233 and the ROI image generation unit 234.

[0208] The write information for each write target ROI is input from the first information extraction unit 23 to the ROI image address generation unit 233 and the ROI image generation unit 234 in the storage order (in ascending order of the starting point X coordinate) in the X-coordinate ordered data. For example, when the first information extraction unit 231 detects the input of the payload data 215A corresponding to the line PL41 (see FIG. 15) of the payload data, it outputs the write information for the write target ROI to the ROI image address generation unit 233 and the ROI image generation unit 234 in the order of the regions of interest ROI2, ROI1. This enables the ROI image address generation unit 233 and the ROI image generation unit 234 to execute the process related to the write process of the restoration process (address generation, division of ROI image row data) in the order of the regions of interest ROI2, ROI1, that is, in the storage order in the X-coordinate ordered data (ascending order of the starting point X coordinate (write start X coordinate)).

[0209] The ROI image address generating unit 233 performs an address generating process in the writing process of the restoration process. Based on the ROI restoration information, the ROI image address generating unit 233 generates an address indicating a write destination of image data of the region of interest (ROI) in the frame memory 250. Based on the input of write information generated from the ROI restoration information from the first information extracting unit 231, the ROI image address generating unit 233 generates an address for writing image data of the region of interest (write target ROI) constituting the ROI pixel row data in the payload data 215A to the frame memory 250.

[0210] In this embodiment, the address generated by the ROI image address generation unit 233 is an address corresponding to a restoration position indicating the relative position of the region of interest in the captured image in the frame memory 250. The ROI image address generation unit 233 calculates an address corresponding to the restoration position of the write target ROI based on the write information of the write target ROI input from the first information extraction unit 231. As described above, the ROI image address generation unit 233 specifies coordinates indicating the write start position of the write target ROI based on the write Y coordinate and the write start X coordinate, and calculates a start address from the write start position.

[0211] As described above, the ROI image address generator 233 calculates consecutive addresses for the number of pixels of the image data of the write target ROI based on the start address and the write size. In this way, the ROI image address generator 233 uses the coordinates of the write start position and the write size to specify the restoration position in the frame memory 250 of the image data of the write target ROI in pixel row units, and generates an address corresponding to the restoration position. Note that the specific method of calculating the address value is appropriately designed according to the allocation of the address area of ​​the frame memory 250.

[0212] As described above, the write information for each write target ROI is input from the first information extraction unit 231 in ascending order of the start point X coordinate (write start X coordinate) to the ROI image address generation unit 233. Therefore, the ROI image address generation unit 233 generates an address for each write target ROI in ascending order of the write start X coordinate.

[0213] The ROI image address generating unit 233 issues an address command including the generated address. The issued address command is transferred to the frame memory 250 via the bus. When issuing the address command, the ROI image address generating unit 233 outputs the write size of the write target ROI, that is, the number of pixels (transfer number) of image data of the write target ROI to be written to the frame memory 250, to the ROI image generating unit 234.

[0214] The ROI image generating unit 234 writes image data of the region of interest in units of payload data to the frame memory 250 based on the ROI restoration information. More specifically, the ROI image generating unit 234 divides and writes the ROI pixel row data in the writing process of the restoration processing. The ROI image generating unit 234 divides the ROI pixel row data in the payload data 215A for each region of interest (write target ROI) based on the ROI restoration information and writes it to the frame memory 250. When the write information generated from the ROI restoration information is input from the first information extracting unit 231, the ROI image generating unit 234 divides the ROI pixel row data in the payload data based on the write information. The ROI image generating unit 234 generates write target image data 234A, which is image data of the write target ROI divided from the ROI pixel row data, and writes the write target image data 234A to the frame memory 250.

[0215] The ROI image generator 234 receives ROI decoded image data 232A obtained by decoding the ROI pixel row data in the payload data 215A from the ROI decoder 232. The ROI decoder 232 decodes the ROI pixel row data (compressed image data 147B) in the payload data 215A received from the ROI data separator 215 to generate ROI decoded image data 232A. In this embodiment, the ROI decoded image data 232A includes a payload header PH and a payload footer PF (see FIG. 15).

[0216] The ROI image generating unit 234 divides the ROI decoded image data 232A into units of the write target ROI based on the write information of the write target ROI input from the first information extracting unit 231. The ROI image generating unit 234 divides the ROI decoded image data 232A into units of the write target ROI based on the write size of the write information. As described above, the write information for each write target ROI is input from the first information extracting unit 231 in ascending order of the start point X coordinate (write start X coordinate). Therefore, the ROI image generating unit 234 divides the image data for each write target ROI in ascending order of the write start X coordinate, that is, in order from the payload header side of the ROI decoded image data 232A. For example, the ROI image generating unit 234 separates image data for each write target ROI from the ROI decoded image data 232A and generates write target image data for each write target ROI.

[0217] For example, when the line PL41 (see FIG. 15) of the 41st line of the payload data is input, the ROI image generating unit 234 divides the ROI decoded image data 232A into 400 pixels each based on the write size (400 pixels) of each of the regions of interest ROI2 and ROI1 that are the write target ROIs. Specifically, the ROI image generating unit 234 divides 400 pixels from the beginning (payload header PH side) of the ROI decoded image data 232A as image data of the region of interest ROI2, and divides the remaining 400 pixels as image data of the region of interest ROI1. The ROI image generating unit 234 separates the divided image data of the region of interest ROI2 from the ROI decoded image data 232A to generate image data 234A to be written for the region of interest ROI2, and similarly generates image data 234A to be written for the region of interest ROI1.

[0218] When the ROI image generating unit 234 generates the write target image data 234A, it issues a data command including the write target image data 234A. The issued data command is transferred to the frame memory 250 via the bus 240. As a result, the write target image data 234A of the write target ROI is written to the frame memory 250. The ROI image generating unit 234 may compress the write target image data 234A and generate a data command with the compressed write target image data 234A. In this case, the write target image data 234A is written to the frame memory 250 in a compressed state.

[0219] In this embodiment, as described above, the number of transfers is input to the ROI image generating unit 234 when the ROI image address generating unit 233 issues an address command. The ROI image address generating unit 233 generates addresses in ascending order of the write start X coordinate of the write target ROI. For this reason, in this embodiment, the address command and the data command related to the same write target ROI are transferred to the frame memory 250 at almost the same timing. That is, in the video receiving device 200a, the issuance of the address command and the data command is synchronized. Therefore, the ROI image generating unit 234 issues a data command in response to the input of the number of transfers from the ROI image address generating unit 233, so that the write target image data 234A of one write target ROI can be written to an address indicating the write destination of the one write target ROI. That is, in the frame memory 250, each pixel of the write target image data 234A of the write target ROI included in the data command is written to each address of the write target ROI indicated by the address command.

[0220] Furthermore, when all image data of the write target ROI is separated from the ROI decoded image data 232A and the generation of the write target image data 234A in payload data units of the write target image data 234A is completed, the ROI image generation unit 234 generates a write end signal and outputs it to the first information extraction unit 231. For example, when the ROI image generation unit 234 separates all image data of the write target ROI from the ROI decoded image data 232A and detects a payload footer PF (see FIG. 15) indicating the end of the image data, the ROI image generation unit 234 outputs the write end signal to the first information extraction unit 231. By receiving the write end signal, the first information extraction unit 231 can recognize that the writing of image data of all the regions of interest in each line of the payload data (payload data 215A) has been completed.

[0221] In this manner, the first information processing unit 230a in the video receiving device 200a according to this embodiment writes image data of the region of interest into the frame memory 250 based on the generated ROI restoration information. The first information processing unit 230a (specifically, the ROI image address generating unit 233) also specifies a restoration position in the frame memory 250 corresponding to the relative position of the region of interest in the captured image based on the ROI restoration information. Furthermore, the first information processing unit 230a (specifically, the ROI image generating unit 234) writes image data of the region of interest (image data to be written 234A) into the restoration position in the frame memory 250. In this manner, the video receiving device 200a can restore the image data of the region of interest to a state that can be displayed on a predetermined display device by executing the restoration process.

[0222] In this embodiment, the first information processing unit 230a of the video receiving device 200a executes an information extraction process and a write process in the restoration process, restores the image data of the attention area to a displayable state, and writes the image data to the frame memory 250, so that the image data can be efficiently displayed on the display device 300 (see FIG. 18). In this embodiment, the first information processing unit 230a is configured as hardware (e.g., a restoration processing circuit) capable of executing the information extraction process and the write process in the restoration process. This allows the video receiving device 200a to speed up the process related to the restoration of the image data of the attention area, and improve the number of image data frames processed per unit time in the transmission system 2, that is, to realize processing at a high frame rate. In addition, the entire video receiving device 200a according to this embodiment may be configured as hardware (e.g., a video receiving circuit). In this case, the receiving unit 210 that receives the transmission data 147A including the embedded data and the payload data is also configured as hardware (e.g., a receiving circuit). This makes it possible to speed up the overall processing of the video receiving device 200a and to further increase the number of image data frames processed per unit time in the transmission system 2, that is, to realize processing at a higher frame rate.

[0223] Here, the frame memory 250 in which the image data of the write target ROI is written will be described. The frame memory 250 can store image data (1 byte) of one pixel in an area indicated by one address, for example. In the case of full-color image data, for example, the data of one pixel is one byte for each RGB element, totaling a memory size of three bytes. For this reason, the frame memory 250 may be provided for each RGB element, and configured to write one pixel (3 bytes) of the full-color image data one byte at a time. At this time, for example, the frame memories 250 for each RGB may each have a common address, and by specifying one common address, it may be possible to write image data for one pixel at the same position in each frame memory.

[0224] In addition, the frame memory 250 can write the contents of the transmission data 147A sent from the video transmission device 100 in one frame, for example, in an area different from the image data. In this embodiment, the contents of the embedded data (EBD data 214A) set in the frame header area of ​​the transmission data 147A are also written to the frame memory 250. For example, the first information extraction unit 231 outputs the EBD data 214A to the ROI image generation unit 234 when writing of image data of all regions of interest (ROI) in the payload data 215A is completed (input of a write end signal). The ROI image generation unit 234 transfers the input EBD data 214A to the frame memory 250. The EBD data 214A is stored in the frame memory as image additional information. As a result, the image data of the region of interest and the embedded data transmitted in one image data frame are stored in the frame memory 250.

[0225] In the video receiving device 200a according to this embodiment, when image data of the region of interest is output to the display device 300, processing is performed using the embedded data stored in the frame memory 250 together with the image data. This allows the display device 300 to display the display data including the image data of the region of interest written in the frame memory 250 in a state that is easier to recognize.

[0226] Furthermore, as will be described in detail later, the image data of the attention area written in the frame memory 250 is read out in frame memory units and output from the video receiving device 200a to the display device 300 (see FIG. 18), and displayed on the screen of the display device 300. Therefore, the image data of the attention area restored in the first information processing unit 230a and written in the frame memory 250 can be displayed on the screen of a predetermined display device while maintaining the relative position in the captured image. This allows the image data of the attention area to be efficiently displayed on the display device 300. The process related to the display of the image data of the eye area ROI read out from the frame memory 250 will be described later.

[0227] Next, the flow of restoration processing in the video receiving device 200a according to this embodiment will be described with reference to Fig. 20. Fig. 20 is a flowchart showing an example of the flow of restoration processing in the video receiving device 200a according to this embodiment.

[0228] (Step S101) When the receiving unit 210 provided in the video receiving device 200a receives a predetermined packet from the video transmitting device 100, it determines whether or not the packet is a packet (SOF) indicating the start of a frame (in this example, an image data frame). When the receiving unit 210 determines that an SOF has been received, it outputs a control signal indicating that an SOF has been received to the first information processing unit 230a. Based on this, the first information processing unit 230a executes the process of step S103. On the other hand, when the receiving unit 210 determines that the received packet is not an SOF, it repeats the process of step S101 at a predetermined interval and waits for reception of an SOF.

[0229] (Step S103) The first information processing unit 230a resets the information (ROI restoration information, etc.) used in the restoration process in the previous image data frame, and moves the process to step S105. The first information processing unit 230a resets the ROI restoration information in the previous image data frame, etc., by deleting the ROI restoration information (X-coordinate order data, Z-coordinate order data) stored in the register or setting a default value in the register.

[0230] (Step S105) The first information processing unit 230a judges whether embedded data has been input. Specifically, the first information extraction unit 231 included in the first information processing unit 230a judges whether embedded data including ROI information (EBD data 214A) has been input from the EBD interpretation unit 214 included in the receiving unit 210. When the first information extraction unit 231 judges that the EBD data 214A has been input, the process proceeds to step S107. On the other hand, when the first information extraction unit 231 judges that the EBD data 214A has not been input, the process of step S105 is repeated at a predetermined interval, and the process of step S107 and subsequent steps is awaited until the EBD data 214A is input.

[0231] (Step S107) The first information extraction unit 231 extracts ROI information (position information and size information) from the embedded data (EBD data 214A) and moves the process to step S109. For example, the first information extraction unit 231 extracts, from the EBD data 214A, start point coordinates as position information corresponding to the regions of interest ROI0 to 2 cut out from the captured image α, and extracts the X-axis length and the Y-axis length as size information.

[0232] (Step S109) The first information extraction unit 231 rearranges the extracted ROI information to generate ROI restoration information (X-coordinate order data, Y-coordinate order data), and moves the process to step S111. Specifically, the first information extraction unit 231 rearranges the first coordinate data and the second coordinate data (see FIG. 16) generated based on the extracted ROI information as described above to generate ROI restoration information (X-coordinate order data, Y-coordinate order data). In addition, the first information extraction unit 231 generates final line determination information in combination with the ROI restoration information. For example, the first information extraction unit 231 rearranges the first coordinate data of the regions of interest ROI0 to 2 in ascending order of the starting point X coordinate to generate X-coordinate order data (see the left side of FIG. 17), and rearranges the second coordinate data of the regions of interest ROI0 to 2 in ascending order of the starting point Y coordinate to generate Y-coordinate order data (see the right side of FIG. 17). Furthermore, the first information extraction unit 231 generates the end point Y coordinate (899) of the region of interest ROI0, which has the largest value among the Y coordinate ordered data, as final line determination information.

[0233] (Step S111) The first information extraction unit 231 determines whether or not a header of payload data has been input. In this embodiment, when the first information extraction unit 231 detects a signal indicating that the payload data 215A has been input in the ROI decoding unit 232, it determines that the header of the payload data has been input, and moves the process to step S113. On the other hand, when the first information extraction unit 231 has not detected a signal indicating that the payload data 215A has been input, it repeats the process of step S111 at predetermined intervals until it detects the input of the signal, that is, until the header of the payload data has been input.

[0234] (Step S113) The first information extraction unit 231 generates write information (write Y coordinate, write start X coordinate, and write size) used for writing image data of the region of interest (write target ROI) from the ROI restoration information, and moves the process to step S115. The first information extraction unit 231 generates a write Y coordinate from the Y coordinate order data, and generates a write start X coordinate and a write size from the X coordinate order data using the generated write Y coordinate. The first information extraction unit 231 outputs the write information generated for each write target ROI to the ROI image address generation unit 233 and the ROI image generation unit 234 in ascending order of the write start X coordinate. Based on the input write information, the ROI image address generation unit 233 executes the process of step S115, and the ROI image generation unit 234 executes the processes of steps S117 and S119.

[0235] (Step S115) Based on the input write information, the ROI image address generation unit 233 calculates an address in the frame memory 250 that corresponds to the restoration position of the ROI to be written. The calculated address is issued as an address command and transferred to the frame memory 250. In addition, the ROI image address generation unit 233 outputs a transfer number indicating the number of pixels of the ROI to be written to the ROI image generation unit 234 in response to the emission of the address command.

[0236] (Step S117) The ROI image generating unit 234 generates the write target image data 234A based on the write information input from the first information extracting unit 231, and moves the process to step S119. The ROI image generating unit 234 divides the ROI decoded image data 232A input from the ROI decoding unit 232 into write target ROI units based on the input write information, and generates the write target image data 234A.

[0237] (Step S119) The ROI image generation unit 234 issues a data command including the write target image data 234A, and writes the write target image data 234A to the frame memory 250 via the bus 240. The ROI image generation unit 234 issues a data command at the timing when the transfer number is input from the ROI image address generation unit 233. The ROI image generation unit 234 also outputs a predetermined signal indicating that the data command has been issued to the first information extraction unit 231. When the first information extraction unit 231 receives the signal, it executes the process of step S121. When the ROI image generation unit 234 detects the payload footer PF of the ROI decoded image data 232A, it also outputs a write end signal to the first information extraction unit 231.

[0238] (Step S121) The first information extraction unit 231 judges whether or not all image data of the write target ROI (ROI pixel row data in the ROI decoded image data 232A) in the input payload data of one line (payload data 215A) has been written to the frame memory 250. The first information extraction unit 231 makes this judgment when a predetermined signal indicating that a data command has been issued is input from the ROI image generation unit 234. When a write end signal is input from the ROI image generation unit 234, the first information extraction unit 231 judges that writing of image data of the write target ROI in the payload data of one line has ended, and moves the process to step S123. On the other hand, when a write end signal is not input, the first information extraction unit 231 judges that image data of the write target ROI that has not been written to the frame memory 250 remains, and returns the process to step S115.

[0239] In this manner, in the restoration process in this embodiment, the processes from step S115 to step S119 are repeated the number of times corresponding to the number of ROIs to be written in each line of the payload data. This allows image data of all ROIs to be written in each line of the payload data to be written in the frame memory 250 in ascending order of the write start X coordinate. For example, all image data of the regions of interest ROI2, ROI0, and ROI1, which are the ROIs to be written in the line PL600 (see FIG. 15) of the payload data, can be written in this order to the frame memory 250.

[0240] (Step S123) The first information extraction unit 231 judges whether or not the last line of the payload data is to be processed. Based on the input of the write end signal, the first information extraction unit 231 compares the value of the write Y coordinate in the current write information with the value of the last line judgment information, and judges that the last line of the payload data is to be processed if the value of the write Y coordinate and the value of the last line judgment information are equal. On the other hand, if the value of the write Y coordinate and the value of the last line judgment information are not equal, the first information extraction unit 231 judges that the last line of the payload data is not to be processed, and returns the process to step S111. This makes it possible to execute the restoration process on the image data of the attention area of ​​the payload data of all lines stored in the packet area R2 in the transmission data 147A.

[0241] 15, first information extraction unit 231 determines that the process is for the last line of payload data when the value of the write Y coordinate is equal to the value of the last line determination information, "899." As a result, in video receiving device 200a, restoration process of image data of the region of interest is performed for all 800 lines, lines PL1 to 800, and image data of regions of interest ROI0 to 2 is written to frame memory 250.

[0242] In this way, the video receiving device 200a and the video transmission system 2 according to this embodiment can perform restoration processing of image data of the region of interest.

[0243] Next, a configuration related to normal image processing in the first information processing unit 230a will be described. The first information processing unit 230a has a normal image address generation unit 235, a normal image decoding unit 236, and a normal image generation unit 237 as components for performing normal image processing. The normal image decoding unit 236 decodes the payload data 215B input from the ROI data separation unit 215 to generate a normal image 224A. The normal image decoding unit 236 decodes the compressed image data 147B included in the payload data 215A and outputs it to the normal image generation unit 237. Furthermore, the normal image address generation unit 235 generates a normal image address for writing the data of the normal image into the frame memory 250 in units of pixel rows, based on the input of a determination result from the ROI data separation unit 215 indicating that the image data included in the payload data is normal image data (payload data 215B including data of the normal image has been output to the normal image decoding unit 236).

[0244] In this embodiment, the frame memory 250 is configured to be able to store image data of the entire region of the captured image (for example, the captured image α). Therefore, the normal image address generating unit 235 may generate addresses for pixel row units (from the left end to the right end in the X-axis direction) of the captured image every time the above-mentioned determination result is input from the ROI data separating unit 215. In this example, when the normal image address generating unit 235 generates an address of data of a normal image corresponding to the first pixel row of the captured image α, it first calculates an address of the frame memory 250 corresponding to the coordinates "0,0(X,Y)" indicating the origin α_о of the captured image α as the first address. Furthermore, after calculating the first address, the normal image address generating unit 235 generates consecutive addresses for storing each pixel of the first pixel row of the captured image α from the write size (1760 pixels) in the X-axis direction in units of pixel rows of the captured image α, similar to the address generation in the ROI image address generating unit 233. Thereafter, the address of the normal image data is generated in the same manner as the address of the first pixel row while incrementing the Y coordinate by 1, and an address command is issued.

[0245] The normal image data may be divided and transferred to the frame memory 250 according to the specifications of the bus 240, etc. In this case, the normal image address generation unit 235 may divide the data into a predetermined number of pixels per pixel row of the captured image to generate and issue address commands. In this case, the normal image address generation unit 235 may output to the normal image generation unit 237 the number of pixels (transfer number) when transferring the normal image data based on the number of divisions of the address.

[0246] The normal image generating unit 237 generates and issues a data command for writing the decoded normal image data input from the normal image decoding unit 236 to the frame memory 250. The normal image generating unit 237 may also divide the normal image data based on the transfer count input from the normal image address generating unit 235, and generate and issue a data command including the divided normal image data. The normal image generating unit 237 may also compress the normal image data and generate a data command with the compressed normal image data. In this case, the compressed normal image data is written to the frame memory 250.

[0247] Moreover, the normal image address generating unit 235 issues a predetermined command to transfer the discrimination result of the data type of the image data input from the ROI data separating unit 215 to the frame memory 250. The discrimination result is stored as image additional information in the frame memory. As a result, the normal image data transmitted in one image data frame and the discrimination result indicating the data type corresponding to the normal image data are stored in the frame memory 250.

[0248] 5. Processing related to display of image data in a video receiving device Next, a description will be given of processing related to the display of image data stored in the frame memory 250. Returning to Fig. 19, the video receiving device 200a according to this embodiment includes a second information processing unit 230b as a configuration for performing processing related to the display of image data stored in the frame memory 250. The second information processing unit 230b executes image data replacement processing, for example, as processing related to the display of image data of the attention area.

[0249] 5-1. Configuration related to image data replacement processing Although the details will be described later, the image data replacement process is a process of replacing non-ROI image data that does not correspond to the image data of the region of interest, among the image data of the display device 300 in units of screens stored in the frame memory 250, with fixed value image data when the image data of the region of interest is displayed on the display device 300. The image data replacement process in the second information processing unit 230b will be described below.

[0250] In this embodiment, for example, when image data (image data of a region of interest (ROI) and normal image data) transmitted in image data frame units from the video transmitting device 100 (see FIG. 18) is written to the frame memory 250, the data (additional information and image data) written based on the image data frame is sequentially read from the frame memory 250 by the DMA controller 260. The read data is output to the second information processing unit 230b.

[0251] The DMA controller 260 in this embodiment controls the transfer of data between the frame memory 250 and the second information processing unit 230b. Note that the DMA controller 260 in this disclosure is not limited to this, and may be configured to control the transfer of data between the frame memory 250 and the first information processing unit 230a.

[0252] The DMA controller 260 is a controller for transferring data by DMA (Direct Memory Access) between the frame memory 250 and the second information processing unit 230b via the bus 240. The DMA controller 260 outputs the additional information read from the frame memory 250 to the second information extraction unit 271. The DMA controller 260 also outputs display target image data RD, which is image data to be displayed read from the frame memory 250, to the image data replacement unit 272. For example, when the DMA controller 260 reads out each pixel of the image data in the frame memory 250 for each address, it outputs the image data to be displayed RD in units of pixel rows in the frame memory 250 to the image data replacement unit 272.

[0253] For example, if the frame memory 250 is capable of storing image data of 1760 pixels (X coordinate = 0 to 1759) per pixel row, the display target image data RD is made up of image data for the 1760 pixels. The display target image data RD output by the DMA controller 260 to the image data replacement section 272 is not limited to pixel row units, and may be, for example, pixel units (each pixel). The DMA controller 260 may also divide the image data in pixel row units into a plurality of display target image data RD and output them to the image data replacement section 272.

[0254] The second information extraction unit 271 in the second information processing unit 230b analyzes the additional information input from the DMA controller 260. Specifically, the second information extraction unit 271 determines whether the input additional information is embedded data (EBD data 214A) or not, and if it is embedded data, generates ROI restoration information based on ROI information (ROI information 120B) included in the embedded data.

[0255] The second information extraction unit 271, like the first information extraction unit 231 in the first information processing unit 230a, extracts position information (start point coordinates) and size information (X-axis length and Y-axis length) of the region of interest from the ROI information (ROI information 120B) included in the embedded data (EBD data 214A) to generate ROI restoration information (X-coordinate order data and Y-coordinate order data). The second information extraction unit 271 stores the generated ROI restoration information in a predetermined register. In addition, the second information extraction unit 271 may output a signal indicating that the ROI restoration information has been generated to the image data replacement unit 272. In this embodiment, the generation of ROI restoration information by the second information extraction unit 271 corresponds to an information extraction process in the image data replacement process.

[0256] When image data of the region of interest is displayed on the display device 300 (see FIG. 18), the image data replacement unit 272 in the second information processing unit 230b replaces non-ROI image data that does not correspond to the image data of the region of interest, among the image data of each screen of the display device 300 stored in the frame memory 250, with fixed value image data. In this embodiment, the replacement of non-ROI image data in the image data replacement unit 272 corresponds to a replacement process in the image data replacement process.

[0257] Here, the fixed value image data is, for example, image data having a fixed aspect. In this embodiment, the fixed value image data is, for example, image data of a fixed color (for example, white). Therefore, when the image data replacement process in this embodiment is executed, the data of the non-ROI image is replaced with white image data. Normally, the image data transmitted in the image data frame is sequentially overwritten in each address area of ​​the frame memory. Furthermore, when writing image data in image data frame units, the frame memory is not usually initialized. Therefore, when image data of the region of interest is newly overwritten in the frame memory, the image data transmitted in the previous image data frame remains in addresses other than the address to be overwritten.

[0258] In the video receiving device 200a according to this embodiment, when image data of the attention area is displayed, image data replacement processing is executed to replace the data of the non-ROI image, i.e., image data remaining in addresses other than the address to be overwritten, with fixed value image data. This enables the video receiving device 200a to improve the visibility of the image data of the attention area to be displayed on the display device 300. Note that when the image data of the attention area is written in the frame memory 250 at a restored position (a position corresponding to a relative position in the captured image), the fixed value image data is displayed on the display device 300 as an image showing the out-of-attention area OE in the captured image.

[0259] When a signal indicating that ROI restoration information has been generated is input from the second information extraction unit 271, the image data replacement unit 272 determines that the image data of the attention region is to be displayed and that the image data of the attention region is included in the display target image data RD input from the DMA controller 260. As a result, the image data replacement unit 272 executes replacement (replacement process) of non-ROI image data for the display target image data RD based on the ROI restoration information generated by the second information extraction unit 271.

[0260] In the replacement process of the image data replacement processing, the image data replacement unit 272 specifies non-ROI image data based on the ROI restoration information generated by the second information extraction unit 271. The image data replacement unit 272 specifies non-ROI image data in the image data in units of screens stored in the frame memory 250 based on the X-coordinate order data and the Y-coordinate order data. Also, the image data replacement unit 272 determines whether each pixel of the image data in units of screens in the frame memory 250 is included in the image data of the region of interest or included in the non-ROI image data based on the ROI restoration information.

[0261] Specifically, the image data replacement unit 272 acquires the coordinate position of each pixel of the image data in units of a screen (image data in the frame memory 250), and determines whether or not the coordinate position is included in the region of interest. For example, when the image data to be displayed RD in units of pixel rows read out from the frame memory 250 is input, the image data replacement unit 272 acquires the coordinate position of each pixel of the image data to be displayed RD. For example, the image data replacement unit 272 acquires the coordinate position of each pixel in sequence from the top (X coordinate=0) of the image data to be displayed RD (for example, X coordinate=0 to 1759).

[0262] In this embodiment, the image data replacement unit 272 acquires the coordinate position of the read image data (display target image data RD) using a counter. In this embodiment, the second information processing unit 230b has a counter unit used for various measurement processes. In this embodiment, the second information processing unit 230b has a horizontal counter Xct and a vertical counter Yct as counter units. The horizontal counter Xct is a counter that measures the X-axis direction position (horizontal position) of the display target image data RD read from the frame memory 250 in the captured image. The counter value of the horizontal counter Xct corresponds to the X coordinate in the captured image (e.g., the captured image α). The vertical counter Yct is a counter that measures the Y-axis direction position (vertical position) of the display target image data RD in the captured image. The value of the vertical counter Yct corresponds to the Y coordinate in the captured image and indicates the position of the pixel row in the captured image. For example, when the values ​​of the horizontal counter Xct and the vertical counter Yct are "0", it indicates the origin α_о(0,0) in the captured image.

[0263] When the image data replacement unit 272 receives the image data RD to be displayed, it refers to the value of the horizontal counter Xct and acquires the coordinates of each pixel of the image data RD to be displayed in order from the top. In this embodiment, the video receiving device 200a performs various processes in the device in synchronization with a predetermined clock signal. Therefore, the counter value of the horizontal counter Xct is updated (incremented) by one in synchronization with the clock signal. In this embodiment, when the counter value of the horizontal counter Xct reaches the end of the X coordinate of the captured image, that is, the value indicating the right end of the captured image, the value of the vertical counter Yct is updated (incremented by one). For example, the X coordinate of the right end of the captured image α is "1759". Therefore, when the counter value of the horizontal counter Xct becomes "1759", the counter value of the horizontal counter Xct is reset to "0". At this time, the value of the vertical counter Yct is updated (incremented by one).

[0264] When the image data replacement unit 272 acquires the coordinate position of each pixel of the image data to be displayed RD, it determines whether or not the acquired coordinate position is included in the region of interest based on the ROI restoration information (X-coordinate order data and Y-coordinate order data). The image data replacement unit 272 outputs image data of pixels included in the region of interest among the pixels of the image data to be displayed RD to the display device 300. On the other hand, the image data replacement unit 272 determines that data of pixels not included in the region of interest among the pixels of the image data to be displayed RD is data of a non-ROI image, replaces the image data of the pixel with fixed value image data, and outputs the image data to the display device 300.

[0265] For example, in order to specify the data of the non-ROI image, the image data replacement unit 272 judges whether or not each of the Y coordinate (pixel Y coordinate) and the X coordinate (pixel X coordinate) in the captured image of each pixel of the display target image data RD is included in the attention area. For example, the image data replacement unit 272 first judges the pixel Y coordinate using the Y coordinate order data of the ROI restoration information.

[0266] Here, an example of pixel Y coordinate determination in the replacement process of the image data replacement processing will be described. The image data replacement unit 272 performs pixel Y coordinate determination using, for example, the counter value of a vertical counter Yct indicating the pixel Y coordinate and the Y coordinate order data. The image data replacement unit 272 extracts second coordinate data in order from the top of the Y coordinate order data (ascending order of the start point Y coordinate), and determines whether or not the counter value of the vertical counter Yct is included within the range from the start point Y coordinate to the end point Y coordinate in the extracted second coordinate data (ROI vertical range) (first determination condition).

[0267] Furthermore, when the image data replacement unit 272 determines that the counter value of the vertical counter Yct is included within the ROI vertical range (the first determination condition is satisfied), it extracts the first coordinate data of the region of interest (ROI) associated with the second coordinate data from the X-coordinate order data, and determines the pixel X-coordinate. On the other hand, when the image data replacement unit 272 determines that the counter value of the vertical counter Yct is not included within the ROI vertical range in any of the second coordinate data in the Y-coordinate order data (the first determination condition is not satisfied), it determines that the pixel being processed in the image data to be displayed RD is not included in the image data of the region of interest, but is included in the non-ROI image data.

[0268] Next, an example of pixel X-coordinate determination in the replacement process of the image data replacement processing will be described. The image data replacement unit 272 extracts from the X-coordinate order data the first coordinate data associated with the second coordinate data when the first determination condition is met, and determines whether or not the counter value of the horizontal counter Xct is included within the range (ROI horizontal range) from the start point X-coordinate to the end point X-coordinate in the extracted first coordinate data (second determination condition). In this way, the image data replacement unit 272 performs the determination of the pixel X-coordinate using, for example, the counter value of the horizontal counter Xct indicating the pixel X-coordinate and the X-coordinate order data.

[0269] When the image data replacement unit 272 determines that the counter value of the horizontal counter Xct is included within the ROI horizontal range (the second judgment condition is satisfied), it determines that the pixel being processed in the image data RD to be displayed is included in the attention area and is included in the image data of the attention area. In other words, when both the first judgment condition and the second judgment condition are satisfied and it is determined that the coordinate position of the pixel to be processed (the pixel to be judged as being within the attention area or not) in the image data RD to be displayed is included in the attention area, the image data replacement unit 272 determines that the pixel being processed is included in the image data of the attention area.

[0270] On the other hand, when the image data replacement unit 272 determines that the counter value of the horizontal counter Xct is not included within the horizontal ROI range (the second judgment condition is not satisfied), it determines that the pixel being processed in the display target image data RD is not included in the image data of the attention region, but is included in the non-ROI image data. In this way, when either the above-mentioned first judgment condition or the second judgment condition is not satisfied and it is determined that the coordinate position of the pixel being processed is not included in the attention region, the image data replacement unit 272 determines that the pixel is included in the non-ROI image data.

[0271] When the image data replacement unit 272 determines in the replacement process of the image data replacement processing that the coordinate position of the pixel being processed in the image data to be displayed RD is included in the attention area, the image data replacement unit 272 outputs (sends) the pixel as image data to be displayed to the display device 300. On the other hand, when the image data replacement unit 272 determines in the replacement process of the image data replacement processing that the coordinate position of the pixel being processed in the image data to be displayed RD is included in the non-ROI image data and not in the attention area, the image data replacement unit 272 replaces the data of the pixel (image data to be displayed) with fixed value image data and outputs (sends) it to the display device 300.

[0272] 5-2. Image data replacement processing method Next, the flow of image data replacement processing in the video receiving device 200a according to this embodiment (image data replacement processing method) will be described with reference to Fig. 22. Fig. 21 is a flowchart showing an example of the flow of image data replacement processing in the video receiving device 200a according to this embodiment.

[0273] (Step S201) In the second information processing unit 230b included in the video receiving device 200a, the second information extraction unit 271 determines whether or not it is the start of an image data frame. In this embodiment, the image data frame in the image data replacement process indicates a series of data read from the frame memory 250 by the DMA controller 26. In other words, a series of read data including additional information and image data sequentially read from the frame memory 250 corresponds to the image data frame in the image data replacement process.

[0274] In step S201, the second information extraction unit 271 determines whether or not the DMA controller 260 has started reading data from the frame memory 250. If the second information extraction unit 271 determines that reading data from the frame memory 250 has started (image data frame has started), for example, by inputting a predetermined signal or command from the DMA controller 260, the process proceeds to step S203. On the other hand, if the second information extraction unit 271 determines that reading data from the frame memory 250 has not started (image data frame has not started), the process of step S201 is repeated at predetermined intervals until the reading is started.

[0275] (Step S203) The second information extraction unit 271 resets various pieces of information generated when the previous image data frame was read in the image data replacement process, and moves the process to step S205. The second information extraction unit 271 resets the ROI restoration information in the previous image data frame, etc., by erasing the ROI restoration information (X-coordinate order data, Z-coordinate order data) stored in the register or setting a default value in the register.

[0276] (Step S205) The second information extraction unit 271 determines whether embedded data has been input. Specifically, the second information extraction unit 271 determines whether embedded data (EBD data 214A) including ROI information as additional information has been input from the DMA controller 260. When the first information extraction unit 231 determines that the EBD data 214A read from the frame memory 250 has been input, the process proceeds to step S207. On the other hand, when the second information extraction unit 271 determines that the EBD data 214A has not been input, the process repeats the process of step S205 at a predetermined interval and waits to execute the processes after step S207 until the EBD data 214A is input.

[0277] When the determination result indicating the data type of the normal image is input as additional information from the DMA controller 260 to the second information extraction unit 271, the second information processing unit 230b executes processing related to display of the data of the normal image. This processing will be described later.

[0278] (Step S207) The second information extraction unit 271 extracts ROI information (position information and size information) from the embedded data (EBD data 214A) in the same manner as the first information extraction unit 231 in step S107 (see FIG. 20) in the restoration process, and moves the process to step S209.

[0279] (Step S209) The second information extraction unit 271 rearranges the extracted ROI information to generate ROI restoration information (data in X-coordinate order, data in Y-coordinate order) in the same manner as the processing of the first information extraction unit 231 in step S109 (see FIG. 20) in the restoration processing, and moves the processing to step S211. At this time, the second information extraction unit 271 stores the ROI restoration information in a register, and outputs a signal indicating that the ROI restoration information has been generated to the image data replacement unit 272. Note that, for example, the register in which the ROI restoration information generated by the second information extraction unit 271 in the image data replacement processing is stored is different from the register in which the ROI restoration information generated by the first information extraction unit 231 in the restoration processing is stored.

[0280] (Step S211) In the second information processing unit 230b provided in the video receiving device 200a, the image data replacement unit 272 determines whether payload data has been input. The payload data in the image data replacement process indicates the display target image data RD in units of pixel rows read for each address from the frame memory 250 by the DMA controller 260. When the display target image data RD has been input from the DMA controller 260, the second information extraction unit 271 determines that payload data has been input and moves the process to step S213. On the other hand, when the display target image data RD has not been input from the DMA controller 260, the second information extraction unit 271 determines that payload data has not been input and repeats the process of step S211 at a predetermined interval (for example, a period synchronized with a clock signal) until the display target image data RD is input.

[0281] Hereinafter, the processes of steps S213, S215, S217, or S219 are executed for all pixels in the payload data (image data to be displayed RD), i.e., for each pixel. Therefore, the image data replacement unit 272 repeats the processes of steps S213 and S215 until the replacement process of the image data replacement process is completed for all pixels in the image data to be displayed RD (YES in step S221), and executes the process of step S217 or S219 depending on the result of step S215.

[0282] (Step S213) The image data replacement unit 272 acquires the image data to be displayed, i.e., the coordinates of the pixel to be displayed in the image data to be displayed RD read out from the frame memory 250, and moves the process to step S215. The image data replacement unit 272 acquires the counter value of the vertical counter Yct as the pixel Y coordinate. The image data replacement unit 272 also acquires the counter value of the horizontal counter Xct as the pixel X coordinate. In this way, the coordinates of each pixel are acquired. In the image data replacement process, the image data replacement unit 272 acquires the coordinates of each pixel in the image data to be displayed RD, starting from the top (X coordinate=0), each time it executes step S213 in a cycle synchronized with the clock signal, for example.

[0283] The image data replacement unit 272 judges whether the coordinate position of the pixel to be displayed acquired in step S213 is included in the region of interest (ROI). As described above, the image data replacement unit 272 judges the value of the vertical counter Yct (pixel Y coordinate) based on the Y coordinate order data, and judges the value of the horizontal counter Xct (pixel X coordinate) based on the X coordinate order data. When the image data replacement unit 272 judges that the pixel Y coordinate is included in the ROI vertical range (first judgment condition is satisfied) and that the pixel X coordinate is included in the ROI horizontal range (second judgment condition is satisfied), it judges that the pixel to be displayed is included in the region of interest (ROI) and moves the process to step S217. On the other hand, when the image data replacement unit 272 judges that either one of the first judgment condition and the second judgment condition is not satisfied, it judges that the pixel to be displayed is not included in the region of interest but is included in non-ROI image data, and moves the process to step S219.

[0284] (Step S217) Based on the determination that the pixel to be displayed is included in the attention area, the image data replacement section 272 outputs the pixel to the display device 300 (see FIG. 18), and the process proceeds to step S221.

[0285] (Step S219) The image data replacement unit 272 replaces the pixel to be displayed with fixed-value image data based on the determination that the pixel is included in the non-ROI image data. Specifically, the image data replacement unit 272 outputs the fixed-value image data to the display device 300 in place of the pixel to be displayed, and proceeds to step S221.

[0286] (Step S221) The image data replacement unit 272 judges whether all image data in the line being processed in the payload data, that is, all pixels in the display target image data RD in units of pixel rows, have been output. When the counter value (pixel X coordinate) of the horizontal counter Xct reaches the value of the end X coordinate (end coordinate in the X-axis direction) corresponding to the right end in the captured image, the image data replacement unit 272 judges that all pixels in the display target image data RD have been output, and moves the process to step S223. At this time, the image data replacement unit 272 resets the counter value of the horizontal counter Xct (sets the counter value to "0"). For example, it is assumed that the X coordinate of the right end of the captured image α is "1759" and the frame memory 250 is capable of storing image data of 1760 pixels (X coordinate = 0 to 1759) per pixel row. In this case, the image data replacement unit 272 judges that all pixels in the display target image data RD have been output when the counter value of the horizontal counter Xct is "1759". As described above, when the horizontal counter Xct is reset, the counter value of the vertical counter Yct is updated (increased by 1), thereby updating the value of the pixel Y coordinate that is common to the image data RD to be displayed in pixel row units.

[0287] On the other hand, if the image data replacement unit 272 determines that the counter value of the horizontal counter Xct is less than the value of the terminal X coordinate in the frame memory 250, the process returns to step S213. As a result, the replacement process in the image data replacement process is executed for all pixels in the display target image data RD in pixel row units.

[0288] (Step S223) The image data replacement unit 272 determines whether or not the processing of the final line of the payload data has been completed, i.e., whether the processing of all image data in the frame memory (processing in the replacement process of the image data replacement processing) has been completed. The image data replacement unit 272 updates (adds 1 to) the value of the vertical counter Yct (counter value). For example, when the value (pixel Y coordinate) of the vertical counter Yct has reached the Y coordinate value corresponding to the bottom end of the captured image, the image data replacement unit 272 determines that the processing of all display target image data RD read out from the frame memory 250 has been completed, and ends the image data replacement processing. At this time, the image data replacement unit 272 resets each counter (sets the counter value to "0") in the order of the horizontal counter Xct and the vertical counter Yct. As a result, both the counter values ​​of the horizontal counter Xct and the vertical counter Yct become "0".

[0289] For example, assume that the Y coordinate of the bottom edge of the captured image α is "989" and the frame memory 250 is capable of storing image data of 989 pixel rows. In this case, when the value of the vertical counter Yct is "989", the image data replacement unit 272 determines that the processing of all image data read from the frame memory 250 has been completed, and ends the image data replacement processing. On the other hand, when the image data replacement unit 272 determines that the counter value of the vertical counter Yct is less than the Y coordinate of the bottom edge of the captured image α, the process returns to step S211. As a result, the replacement process in the image data replacement processing is executed for all pixels in the display target image data RD for all pixel rows in the frame memory 250, that is, for all image data in the frame memory 250.

[0290] As described above, the video receiving device 200a (see FIG. 19) according to this embodiment includes a receiving unit 210 that receives a transmission signal including image data of a predetermined number of regions of interest (ROI) cut out from a predetermined captured image in payload data and ROI information in embedded data, and an information processing unit 230 (specifically, a first information processing unit 230a) that can execute a restoration process to restore image data of the regions of interest to a state that can be displayed on the display device 300 (see FIG. 18) based on the ROI information corresponding to the predetermined number of regions of interest extracted from the embedded data. This allows the image data of the regions of interest to be efficiently displayed on the display device 300.

[0291] Furthermore, when image data of the attention area is displayed on the display device 300, the information processing unit 230 (specifically, the second information processing unit 230b) in the video receiving device 200a according to this embodiment replaces non-ROI image data that does not correspond to the image data of the attention area among the image data of the display device 300 in units of screens stored in the frame memory 250 with fixed value image data. This makes it possible to improve the visibility of the image data of the attention area to be displayed on the display device 300.

[0292] In the video receiving device 200a according to this embodiment, the receiving unit 210 and the information processing unit 230 (first information processing unit 230a, second information processing unit 230b) are configured as hardware. This allows the video receiving device 200a to speed up the process related to the restoration of image data of the region of interest in the first information processing unit 230a and the process of replacing image data in the second information processing unit 230b, thereby improving the number of image data frames processed per unit time.

[0293] 5-3. Processing related to displaying normal image data Next, an overview of the process (normal image display process) related to displaying normal image data in the second information processing unit 230b will be described. The second information processing unit 230b executes the normal image display process when a determination result indicating the data type of the normal image is input as additional information from the DMA controller 260 to the second information extraction unit 271.

[0294] Specifically, when the input additional information is a discrimination result indicating a data type corresponding to data of a normal image, the second information extraction unit 271 outputs the discrimination result to the image data replacement unit 272. When the discrimination result is input, the image data replacement unit 272 determines that the display target image data RD input from the DMA controller 260 is data of a normal image. In this case, the image data replacement unit 272 outputs the display target image data RD including the data of the normal image to the display device 300 without executing the image data replacement process. For example, the image data replacement unit 272 may divide the display target image data RD including the data of the normal image into pixel units (one pixel at a time) and output them to the display device 300, or may output them in pixel row units. In this way, the image data replacement unit 272 has a function as an output unit that outputs image data to the display device 300.

[0295] 6. Modifications of the restoration process 6-1. Variation 1 In the present embodiment, the ROI image generator 234 that executes the writing process in the restoration process divides the ROI pixel row data for each region of interest (ROI) and writes the divided data to the frame memory 250, but the present disclosure is not limited to this. Various methods are conceivable for writing the ROI pixel row data in the payload data 215A to the frame memory.

[0296] For example, the ROI image generating unit 234 in the first information processing unit 230a may transfer and write image data (ROI pixel row data) of the region of interest in the payload data 215A to the frame memory 250 by burst transfer based on the ROI restoration information. For example, when a predetermined condition is satisfied in the writing process, the ROI image generating unit 234 inserts dummy image data between image data of two or more regions of interest in the payload data, and generates dummy insertion image data 234B in which the image data of the two or more regions of interest are concatenated. Then, for example, the ROI image generating unit 234 may write the generated dummy insertion image data 234B to the frame memory continuously by one instruction (for example, issuance of one data command).

[0297] That is, the ROI image generating unit 234 may transfer the generated dummy insertion image data 234B to the frame memory 250 by burst transfer, and continuously write the dummy insertion image data 234B to the frame memory 250. Here, the dummy image data included in the dummy insertion image data 234B may be, for example, predetermined image data (for example, image data similar to the fixed value image data in the image data replacement process) that fills a gap between two or more regions of interest.

[0298] In this modification, for example, when image data of two ROIs to be written is included in one ROI pixel row data in the payload data 215A, the first information extraction unit 231 may determine whether the above-mentioned burst transfer can be performed according to the distance between the image data of the ROIs to be written, that is, the length of the gap. For example, the first information extraction unit 231 may determine that burst transfer can be performed when the gap between the image data of the region of interest (ROI to be written) to be written is narrow and the memory size (number of bytes) of the dummy image data to be inserted into the gap is within a predetermined size (one example of a predetermined condition). In this case, the first information extraction unit 231 generates a dummy image size in addition to the write size of each image data of the ROI to be written as write information based on the ROI restoration information.

[0299] For example, when the memory size of the dummy image data is within 128 bytes, the first information extraction unit 231 may determine that burst transfer is executable and generate the dummy image size as write information. The memory size of the dummy image data used to determine whether burst transfer is executable and the size of image data that can be burst transferred are appropriately selected according to, for example, the specifications of the bus 240 used to transfer image data in the video receiving device 200a. When the memory size of the dummy image data exceeds 128 bytes and burst transfer is impossible, the first information extraction unit 231 may determine to write image data for each write target ROI as in the above embodiment.

[0300] When the write information including the dummy image size is input from the first information extraction unit 231, the ROI image generation unit 234 generates dummy image data having the dummy image size, and inserts it between the image data of, for example, two write target ROIs to generate dummy insertion image data 234B. At this time, the ROI image generation unit 234 may determine the insertion position of the dummy image data based on the write size (division size) of each attention area ROI. When the ROI image generation unit 234 generates the dummy insertion image data 234B, it issues a data command including the generated dummy insertion image data 234B, and writes the image data of multiple (for example, two) write target ROIs to the frame memory 250 by burst transfer.

[0301] When the ROI image generation unit 234 performs burst transfer of image data of two or more write target ROIs, the size of the image data transferred to the frame memory 250 by one data command (burst transfer size), i.e., the image size of the dummy insertion image data 234B, is the sum of the write sizes of the image data of the write target ROIs that are the subject of burst transfer plus the size of the dummy image data (dummy image size).

[0302] Furthermore, when the first information extraction unit 231 determines that burst transfer is to be performed for image data of a plurality of (e.g., two) write target ROIs, the ROI image address generation unit 233 generates an address of the frame memory 250 in burst transfer units based on write information including the dummy image size. When burst transfer of the dummy insertion image data 234B is performed, the ROI image address generation unit 233 generates an address indicating the write destination of the dummy insertion image data 234B in the frame memory 250 based on the ROI restoration information. Specifically, the ROI image address generation unit 233 generates an address indicating the write destination of the dummy insertion image data 234B based on the write information including the dummy image size generated by the first information extraction unit 231.

[0303] In this case, the ROI image address generating unit 233 generates the start address of the dummy insertion image data 234B based on the write start X coordinate and write Y coordinate included in the write information. Furthermore, the ROI image address generating unit 233 adds to the start address the write size of the dummy insertion image data 234B (the sum of the write sizes of the image data of the write target ROI plus the dummy image size). This generates consecutive addresses that are the write destinations for each dummy insertion image data.

[0304] Furthermore, the ROI image address generation unit 233 outputs the image size (number of pixels) of the dummy insertion image data 234B written by burst transfer as the burst transfer number to the ROI image generation unit 234. When the burst transfer number is input, the ROI image generation unit 234 issues a data command including write dummy insertion image data 234B of a size according to the burst transfer number. In this way, the dummy insertion image data 234B is transmitted to the frame memory 250 by burst transfer and written therein.

[0305] Also, when the payload data 215A includes three or more write target ROIs and there are two or more gaps between the regions of interest, the first information extraction unit 231 may determine the memory size of the dummy image data for each of the gaps. When the first information extraction unit 231 determines that the dummy insertion image data 234B including the image data of all the write target ROIs in the payload data 215A can be transferred to the frame memory 250 (when the dummy image sizes related to all the gaps are equal to or smaller than a predetermined number), the dummy insertion image data 234B including the image data of all the write target ROIs is written to the frame memory 250 by issuing one data command. In other words, the number of times of writing to the frame memory 250 is one.

[0306] On the other hand, for example, assume that a wide gap in which the dummy image size exceeds a predetermined number (for example, 128 bytes) is included in multiple gaps between image data of the write target ROIs (for example, three gaps between image data of four write target ROIs). In this case, the first information extraction unit 231 may determine to perform burst transfer by dividing the ROI pixel row data in the payload data 215A into before and after the wide gap. For example, dummy insertion image data 234B-1 including image data of the first two write target ROIs out of the four write target ROIs and dummy insertion image data 234B-2 including image data of the remaining (third and fourth from the beginning) write target ROIs may be generated. In this case, the number of times of writing to the frame memory 250 is two.

[0307] In addition, the first information extraction unit 231 may determine that, for image data of multiple write target ROIs in the payload data 215A, a predetermined number of write target ROIs (2 or more) are burst transferred, and the image data of the remaining write target ROIs is written in units of interest as usual.

[0308] In this way, by performing burst transfer in ROI pixel row data units, the number of writes can be reduced and the write process can be speeded up compared to when image data is written to the frame memory in region of interest (ROI) units.

[0309] 6-2.Variation 2 In the present embodiment, the address generated by the ROI image address generating unit 233 is an address corresponding to a restored position indicating the relative position of the region of interest in the captured image in the frame memory 250, but the present disclosure is not limited to this. For example, the ROI image address generating unit 233 may be configured to generate an address indicating an arbitrary position (an example of a predetermined position) where image data of a predetermined number of regions of interest can be displayed without overlapping on the screen of the display device 300. In other words, the ROI image generating unit 234 may be configured to write image data of the region of interest so that image data of a predetermined number of regions of interest can be displayed without overlapping on the screen of the display device 300.

[0310] For example, the ROI image address generating unit 233 may generate an address indicating an arbitrary position (Y-axis arbitrary position) where image data of a predetermined number of regions of interest can be displayed sequentially in the Y-axis direction of the display device 300 so as not to overlap each other. The ROI image generating unit 234 may be configured to write image data of the regions of interest at an arbitrary position on the Y-axis within the screen of the display device 300. In this case, the first information extracting unit 231 uses the X-coordinate order data and the Y-coordinate order data of the ROI restoration information to generate write information capable of generating an address according to the Y-axis arbitrary position. For example, the first information extracting unit 231 uses the X-coordinate order data and the Y-axis coordinate order data (see FIG. 17) to generate write information used for address generation for displaying the regions of interest ROI0 to 2 (see FIG. 13) at arbitrary positions on the Y-axis on the display device 300.

[0311] For example, the first information extraction unit 231 determines the display order of the regions of interest ROI0-2 on the display device 300 based on the Y-axis coordinate order data. For example, the first information extraction unit 231 determines the storage order of the Y-axis coordinate order data, i.e., the ascending order of the starting point Y coordinate, as the display order of the regions of interest ROI0-2. In this example, the order of the regions of interest ROI2, ROI1, ROI0 becomes the display order in the Y-axis direction on the screen of the display device 300.

[0312] After determining the display order, the first information extraction unit 231 determines, for example, a display start Y coordinate, which is the leading Y coordinate when displaying image data of each region of interest (ROI) on the display device 300. As described above, image data for each screen of the display device 300 is written to the frame memory 250. Therefore, the display position (coordinates) on the screen of the display device 300 corresponds to the write position in the frame memory 350.

[0313] For example, the image data of the region of interest ROI2, which is the first in the display order, is displayed from the first pixel row of the display device 300. Therefore, the first information extraction unit 231 sets the write start Y coordinate of the image data of the region of interest ROI2 to "0". Furthermore, the image data of the region of interest ROI1 is displayed below the image data of the region of interest ROI2 so as not to overlap with the image data of the region of interest ROI2. Therefore, the first information extraction unit 231 sets the display start Y coordinate of the image data of the region of interest ROI1 to "600 (= 0 + 600 (length of the region of interest ROI2 in the Y-axis direction)". Note that the length of the Y-axis direction is calculated as "end point Y coordinate - start point Y coordinate" in the Y-coordinate order data. Similarly, the first information extraction unit 231 sets the display start Y coordinate of the image data of the region of interest ROI0, which is displayed below the image data of the region of interest ROI1, to "1200 (= 600 (display start Y coordinate of the image data of the region of interest ROI1) + 600 (length of the region of interest ROI1 in the Y-axis direction)".

[0314] After determining the display start Y coordinate of the image data of each attention area, the first information extraction unit 231 updates the display start Y coordinate value corresponding to each attention area by adding 1 each time the image data of each attention area is input as payload data. This makes it possible to generate the display start Y coordinate corresponding to an arbitrary Y-axis position.

[0315] Also, a display start X coordinate, which is the leading X coordinate when image data of the attention area is displayed on the display device 300 at an arbitrary Y-axis position on the screen of the display device 300, is determined. In this modified example, the write start X coordinate of the image data of each attention area is a constant value. For example, when the image data of each attention area is displayed at an arbitrary Y-axis position, the image data of each attention area is displayed in a left-justified state on the screen of the display device 300. In this case, the first information extraction unit 231 generates a common value "0" for each attention area as the display start X coordinate. In other words, when the image data of the attention area is displayed at an arbitrary Y-axis position, the image data of each attention area is displayed in a left-justified state starting from the upper left end (origin position) of the screen of the display device 300, lined up in the Y-axis direction in order from the top end of the screen. Also, the first information extraction unit 231 generates a write size in the same manner as in the above embodiment.

[0316] When the write information corresponding to the display of the image data of the region of interest at an arbitrary position on the Y axis is input from the first information extraction unit 231, the ROI image address generation unit 233 generates an address for writing the image data of the region of interest in the frame memory 250 based on the write information. The ROI image address generation unit 233 can generate an address in the frame memory 250 corresponding to the arbitrary position on the Y axis by generating an address based on the input write information in the same manner as in the above embodiment. Also, the ROI image generation unit 234 generates the image data to be written 234A and issues a data command at a timing synchronized with the issuance of the address by the ROI image address generation unit 233 in the same manner as in the above embodiment. This allows the ROI image generation unit 234 to write image data of the region of interest (ROI) at an arbitrary position on the Y axis where a predetermined number of image data of the region of interest can be displayed sequentially in the Y axis direction without overlapping on the screen of the display device 300.

[0317] Note that this modification is not limited to this. For example, the ROI image address generating unit 233 may generate an address indicating an arbitrary position (X-axis arbitrary position) where image data of a predetermined number of attention areas can be displayed sequentially in the X-axis direction of the display device 300 so as not to overlap each other. Also, the ROI image generating unit 234 may be configured to write image data of the attention area to an arbitrary X-axis position within the screen of the display device 300. In this case, the first information extracting unit 231 uses the X-coordinate order data and the Y-coordinate order data of the ROI restoration information to generate write information that can generate an address according to the arbitrary X-axis position. At this time, the first information extracting unit 231 determines the display order of each attention area as the storage order of the X-coordinate order data, that is, the ascending order of the starting point X coordinate (the order of attention area RI2, ROI0, ROI1).

[0318] When image data of the attention area is displayed at an arbitrary position on the X axis, the image data of each attention area is displayed sequentially from the top left corner of the screen of the display device 300 toward the X axis in order from the left edge of the screen.

[0319] After determining the display order, the first information extraction unit 231 determines the display start X coordinate. For example, the region of interest ROI2 which is the first in the display order is displayed from the upper left corner position of the display device 300. Therefore, the first information extraction unit 231 sets the display start X coordinate of the region of interest ROI2 to "0". In addition, the region of interest ROI0 is displayed to the right of the region of interest ROI2 so as not to overlap with the region of interest ROI2. Therefore, the first information extraction unit 231 sets the write start X coordinate of the region of interest ROI0 to "400 (= 0 + 400 (length of region of interest ROI2 in the X-axis direction)". Note that the length in the X-axis direction is calculated as "end point X coordinate - start point X coordinate" in the X-coordinate ordered data. Similarly, the first information extraction unit 231 sets the display start X coordinate of region of interest ROI1 displayed to the right of region of interest ROI0 to "800 (= 400 (display start X coordinate of the image data of region of interest ROI0) + 400 (length of region of interest ROI0 in the X-axis direction)".

[0320] Moreover, when image data of the attention area is displayed at an arbitrary position on the X axis on the screen of the display device 300, the display start Y coordinate of the image data of each attention area is a constant value. For example, when image data of the attention area is displayed at an arbitrary position on the X axis, the image data of each attention area is displayed aligned from the top on the screen of the display device 300. In this case, the first information extraction unit 231 generates a common value "0" for each attention area as the display start Y coordinate. Moreover, the first information extraction unit 231 updates the value of the display start Y coordinate corresponding to each attention area by adding 1 each time the image data of each attention area is input as payload data. This makes it possible to generate the display start Y coordinate corresponding to the arbitrary position on the X axis. Moreover, the first information extraction unit 231 generates the write size in the same manner as in the above embodiment.

[0321] When write information corresponding to display of image data of the region of interest at an arbitrary position on the X axis is input from the first information extraction unit 231, the ROI image address generation unit 233 generates an address based on the input write information, as in the above embodiment. This makes it possible to generate an address in the frame memory 250 corresponding to the arbitrary position on the X axis. Furthermore, the ROI image generation unit 234 can write image data of the region of interest at an arbitrary position on the X axis where a predetermined number of image data of the region of interest can be displayed sequentially in the X axis direction without overlapping on the screen of the display device 300.

[0322] In this way, by executing the restoration process in this modified example, the video receiving device 200a can restore the image data of the region of interest (ROI) to a state in which it can be displayed at any position (for example, any position on the X-axis or any position on the Y-axis) on the display device 300. This allows the image data of the region of interest to be efficiently displayed on the display device 300 (see FIG. 18). Also, by displaying the image data of a predetermined number of regions of interest aligned in one direction (the X-axis or Y-axis direction) without overlapping each other, it is possible to efficiently perform, for example, recognition processing and comparison processing of the image data of each region of interest.

[0323] 6-3.Other modifications In the above embodiment, the first information extraction unit 231 of the first information processing unit 230a of the video receiving device 200a performs the information extraction process in the restoration process (generation of ROI restoration information from extracted ROI information), but the present disclosure is not limited to this. For example, in the video receiving device 200a according to the present disclosure, the EBD interpretation unit 214 of the receiving unit 210 may extract ROI information (position information and size information) from embedded data in the information extraction process. In this case, the EBD interpretation unit 214 may output the extracted ROI information to the first information extraction unit 231. Furthermore, the first information extraction unit 231 may generate ROI restoration information based on the input ROI information.

[0324] In addition, in the above embodiment, the attention area in the captured image is rectangular as an example, but the present disclosure is not limited to this. Even if the attention area ROI is not rectangular, the video receiving device 200a can execute the restoration process and the image data replacement process in the above embodiment.

[0325] The present disclosure has been described above by citing the base technology, the embodiment, and the modified examples thereof, but the present disclosure is not limited to the above embodiment, etc., and various modifications are possible. Note that the effects described in this specification are merely examples. The effects of the present disclosure are not limited to the effects described in this specification. The present disclosure may have effects other than the effects described in this specification.

[0326] Furthermore, for example, the present disclosure can have the following configuration. (1) A receiver that receives a transmission signal including payload data of image data of a predetermined number of ROIs (Regions of Interest) cut out from a predetermined captured image and including embedded data of ROI information corresponding to the predetermined number of ROIs; A processing unit capable of executing a restoration process for restoring the image data of the ROI to a state that can be displayed on a predetermined display device based on the ROI information extracted from the embedded data; A receiving device comprising: (2) The processing unit extracts position information and size information of each of a predetermined number of the ROIs in the captured image as the ROI information. A receiving device as described in (1) above. (3) The processing unit rearranges the extracted position information and size information to generate ROI restoration information used for restoring image data of the ROI. A receiving device as described in (2) above. (4) The position information includes an X coordinate and a Y coordinate indicating an upper left end portion of the ROI in the captured image, The processing unit generates, as the ROI restoration information, X-coordinate order data in which the position information and the size information are rearranged in ascending order of the X-coordinate values, and Y-coordinate order data in which the position information and the size information are rearranged in ascending order of the Y-coordinate values. A receiving device as described in (3) above. (5) A frame memory capable of storing image data of the ROI is provided, The processing unit writes image data of the ROI into the frame memory based on ROI restoration information generated from the ROI information. A receiving device according to any one of (1) to (4) above. (6) The processing unit writes the image data of the ROI to the frame memory in units of the payload data based on the ROI restoration information. A receiving device as described in (5) above. (7) The processing unit divides the image data of the ROI in the payload data for each ROI based on the ROI restoration information, and writes the divided image data to the frame memory. A receiving device as described in (6) above. (8) When a predetermined condition is satisfied, the processing unit generates dummy-inserted image data by inserting dummy image data between image data of two or more ROIs in the payload data and connecting the image data of the two or more ROIs, and writes the generated dummy-inserted image data continuously to the frame memory in response to a single instruction. A receiving device as described in (6) above. (9) The processing unit specifies a restoration position in the frame memory corresponding to a relative position of the ROI in the captured image based on the ROI restoration information, and writes image data of the ROI to the restoration position. A receiving device according to any one of (5) to (8) above. (10) The processing unit writes the image data of the ROIs into the frame memory so that image data of a predetermined number of the ROIs can be displayed without overlapping on a screen of the display device based on the ROI restoration information. A receiving device according to any one of (5) to (8) above. (11) The processing unit generates an address indicating a write destination of the image data of the ROI in the frame memory based on the ROI restoration information. A receiving device according to any one of (5) to (10) above. (12) The processing unit generates an address indicating a write destination of the dummy insertion image data in the frame memory based on the ROI restoration information. A receiving device as described in (8) above. (13) When the image data of the ROI is displayed on the display device, the processing unit replaces non-ROI image data that does not correspond to the image data of the ROI among the image data of the screen unit of the display device stored in the frame memory with fixed value image data. A receiving device according to any one of (5) to (12) above. (14) The processing unit identifies the non-ROI image data based on the ROI restoration information. The receiving device according to (13) above. (15) The processing unit includes: Based on the ROI restoration information, it is determined whether each pixel of the image data in the screen unit is included in the ROI image data or the non-ROI image data. The receiving device according to (14) above. (16) A transmitter that transmits image data of a region of interest (ROI) as payload data and transmits ROI information as embedded data; The receiving device includes a receiving unit that receives a transmission signal including image data of a predetermined number of ROIs cut out from a predetermined captured image in payload data and ROI information corresponding to the predetermined number of ROIs in embedded data, and a processing unit that is capable of executing a restoration process to restore the image data of the predetermined number of ROIs to a state that can be displayed on a predetermined display device based on the ROI information extracted from the embedded data. Transmission system. (17) The receiving device includes: The processing unit rearranges the position information and size information of each of the predetermined number of ROIs in the captured image extracted as the ROI information, to generate ROI restoration information to be used for restoring the image data of the ROIs. A transmission system as described in (16) above. (18) The receiving device includes a frame memory capable of storing image data of the ROI, and the processing unit writes the image data of the ROI to the frame memory based on the ROI restoration information. A transmission system as described in (17) above. (19) When the image data of the ROI is displayed on the display device, the receiving device replaces non-ROI image data that does not correspond to the image data of the ROI among the image data of the screen unit of the display device stored in the frame memory with fixed value image data in the processing unit. A transmission system as described in (18) above. [Explanation of symbols]

[0327] 1,2 Video transmission system 31,110 Imaging section 32,41 Control section 42 Image Processing Department 100 Video transmission device 100A CSI Transmitter 100B CCI Slave 111 Captured images 112,112a1,112a2,112a3,112a4,112b1,112b4,123a4,223A ROI image 112b Compressed image data 113,114 Location information 115 Priority 116,116a1,116a2 Transmission image 118 images 120,130 Image processing unit 120A, 120A1, 120A2, 130A, 147B Compressed image data 120B ROI information 120C Frame Information 121 ROI cutout 122 ROI Analysis Department 123 Detection unit 124 Priority setting section 125,131 Encoding section 126 Image processing control section 140 Transmitter 141 LINK control section 142 ECC generator 143 PH generation section 144 EBD Buffer 145 ROI data buffers 146 Normal image data buffer 147 Synthesis Section 147A Transmission Data 200, 200a Video receiving device 200A CSI Receiver 200B CCI Master 210 Receiving unit 211 Header Separator 212 Header Interpretation 213 Payload Separation Section 214 EBD Interpretation Division 214A EBD Data 215 ROI Data Separation Unit 215A, 215B payload data 220 Information Processing Department 221 Information extraction section 221A Extract information 222, 232 ROI Decoder 222A Image data 223 ROI Image Generation Unit 224 Normal image decoding section 224A Normal Image 230 Information Processing Department 230a First information processing unit 230b Second information processing section 231 First Information Extraction Department 232A ROI decoded image data 233 ROI image address generation unit 234 ROI Image Generation Unit 234A Image data to be written 235 Normal image address generator 236 Normal Image Decoder 237 Normal Image Generation Unit 240 Bus 250 Frame Memory 260 DMA Controller 271 Second information extraction section 272 Image Data Replacement Section 300 display device Cb color difference component CCI Camera Control Interface CL Clock Lane RD Image data to be displayed

Claims

1. a receiving unit for receiving a transmission signal including, in payload data, image data of a predetermined number of ROIs (Region Of Interests) cut out from a predetermined captured image, and including, in embedded data, ROI information corresponding to the predetermined number of ROIs; A frame memory capable of storing image data of the ROI; a processing unit capable of executing a restoration process for restoring the image data of the ROI to a state that can be displayed on a predetermined display device based on the ROI information extracted from the embedded data; Equipped with The processing unit includes: writing image data of the ROI into the frame memory in units of the payload data based on ROI restoration information generated from the ROI information; When a predetermined condition is satisfied, dummy image data is inserted between image data of two or more ROIs in the payload data to generate dummy inserted image data by connecting the image data of the two or more ROIs, and the generated dummy inserted image data is continuously written to the frame memory by a single instruction; The predetermined condition is that the gap between the image data of the two or more ROIs is narrow, and the memory size of the dummy image data to be inserted into the gap is within a predetermined size. Receiving device.

2. The processing unit extracts position information and size information of each of a predetermined number of the ROIs in the captured image as the ROI information.

2. The receiving device according to claim 1.

3. The processing unit rearranges the extracted position information and size information to generate the ROI restoration information used for restoring image data of the ROI.

3. The receiving device according to claim 2.

4. the position information includes an X coordinate and a Y coordinate indicating an upper left end portion of the ROI in the captured image; The processing unit generates, as the ROI restoration information, X-coordinate order data in which the position information and the size information are rearranged in ascending order of the X-coordinate values, and Y-coordinate order data in which the position information and the size information are rearranged in ascending order of the Y-coordinate values.

4. The receiving device according to claim 3.

5. The processing unit divides the image data of the ROI in the payload data for each ROI based on the ROI restoration information, and writes the divided image data into the frame memory.

2. The receiving device according to claim 1.

6. The processing unit specifies a restoration position in the frame memory that corresponds to a relative position of the ROI in the captured image based on the ROI restoration information, and writes image data of the ROI to the restoration position.

2. The receiving device according to claim 1.

7. The processing unit writes the image data of the ROIs into the frame memory so that image data of a predetermined number of the ROIs can be displayed without overlapping on a screen of the display device based on the ROI restoration information.

2. The receiving device according to claim 1.

8. The processing unit generates an address indicating a write destination of the image data of the ROI in the frame memory based on the ROI restoration information.

2. The receiving device according to claim 1.

9. The processing unit generates an address indicating a write destination of the dummy insertion image data in the frame memory based on the ROI restoration information.

2. The receiving device according to claim 1.

10. The processing unit replaces non-ROI image data that does not correspond to the image data of the ROI, among the image data of the screen unit of the display device stored in the frame memory, with fixed value image data when the image data of the ROI is displayed on the display device.

2. The receiving device according to claim 1.

11. The processing unit identifies the non-ROI image data based on the ROI reconstruction information.

11. The receiving device according to claim 10.

12. The processing unit includes: It is determined whether each pixel of the image data of the screen unit is included in the ROI image data or the non-ROI image data based on the ROI reconstruction information.

12. The receiving device according to claim 11.

13. A transmitting device that transmits image data of a region of interest (ROI) as payload data and transmits ROI information as embedded data; a receiving device including a receiving unit that receives a transmission signal including payload data of image data of a predetermined number of ROIs cut out from a predetermined captured image and embedded data of ROI information corresponding to the predetermined number of ROIs, a frame memory that can store the image data of the ROI, and a processing unit that can execute a restoration process to restore the image data of the predetermined number of ROIs to a state that can be displayed on a predetermined display device based on the ROI information extracted from the embedded data; Equipped with The receiving device, in the processing unit, writing image data of the ROI into the frame memory in units of the payload data based on ROI restoration information generated from the ROI information; When a predetermined condition is satisfied, dummy image data is inserted between image data of two or more ROIs in the payload data to generate dummy inserted image data by connecting the image data of the two or more ROIs, and the generated dummy inserted image data is continuously written to the frame memory by a single instruction; The predetermined condition is that the gap between the image data of the two or more ROIs is narrow, and the memory size of the dummy image data to be inserted into the gap is within a predetermined size. Transmission system.

14. The receiving device includes: The processing unit rearranges the position information and size information of each of the predetermined number of ROIs in the captured image extracted as the ROI information, and generates the ROI restoration information used for restoring the image data of the ROI. The transmission system according to claim 13.

15. When the image data of the ROI is displayed on the display device, the receiving device replaces non-ROI image data that does not correspond to the image data of the ROI, among the image data of the screen unit of the display device stored in the frame memory, with fixed value image data in the processing unit. The transmission system according to claim 13.

Citation Information

Patent Citations

  • Image data control device

    JP2012209831A

  • Image processing device, method thereof, and program

    JP2013164834A

  • Video decoder, video transmission reception system, video decoding method and video transmission reception method

    JP2014039219A

  • Image processing apparatus, image processing method, and image processing system

    JP2016201756A

  • Advanced region-of-interest capabilities for image sensors

    JP2016504890A