System, camera, device, and method

Abstract

To provide a new solution for the need to simultaneously display different regions in an image captured by one camera.SOLUTION: A system includes a device and a camera connected by wire or wirelessly to the device. The device alternately transmits a first request to cause the camera to transmit image data corresponding to a first region, and a second request to cause the camera to transmit image data corresponding to a second region, to the camera. The camera generates first image data corresponding to the first region from overall image data in accordance with the first request, transmits first data including the first image data to the device, generates second image data corresponding to the second region from the overall image data in accordance with the second request, and transmits second data including the second image data to the device. The device outputs a synthesized image including a first image based on the first image data and a second image based on the second image data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to systems, cameras, devices, and methods. [Background technology]

[0002] For example, Japanese Patent Application Laid-Open No. 2012-84968 (Patent Document 1) discloses a configuration for transmitting image data from a camera body to a monitor device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-84968 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a need to simultaneously output images of different regions in an image captured by a single camera, and the present disclosure provides a new solution to this need. [Means for solving the problem]

[0005] (Configuration 1) A system according to one embodiment includes a device and a camera connected to the device via a wired or wireless connection. The device alternately transmits to the camera a first request to cause the camera to transmit image data corresponding to a first region of the camera's imaging area and a second request to cause the camera to transmit image data corresponding to a second region of the camera's imaging area. In accordance with the first request, the camera generates first image data corresponding to the first region from overall image data corresponding to the imaging area and transmits the first data including the first image data to the device. In accordance with the second request, the camera generates second image data corresponding to the second region from the overall image data and transmits the second data including the second image data to the device. The device outputs a composite image including a first image based on the first image data included in the received first data and a second image based on the second image data included in the received second data.

[0006] According to configuration 1, the camera transmits to the device first image data corresponding to the first region based on the first request and second image data corresponding to the second region based on the second request. Since the data transmitted from the camera to the device does not include image data that is unnecessary for outputting the composite image, it is possible to output a composite image that includes images corresponding to at least two regions while suppressing the amount of data transmitted.

[0007] For example, if a camera transmits overall image data to a device and the device extracts image data necessary for a composite image from the overall image data, information about the areas not extracted from the overall image data becomes unnecessary. Sending overall image data that includes such potentially unnecessary information results in wasted communication bandwidth between device 100 and camera 200. Configuration 1 can reduce such wasted communication bandwidth.

[0008] (Configuration 2) In configuration 1, the first request may include first identification information, and the first data may include second identification information corresponding to the first identification information. The device may place the first image at a first position in the composite image based on the received first data including the second identification information, and may place the second image at a second position in the composite image based on the received second data not including the second identification information.

[0009] According to configuration 2, the device can distinguish at least the first image data and the second image data based on whether the first data received from the camera includes second identification information, and determine the position where the image should be output in the composite image. For example, if the content of the area setting is transmitted to the device instead of the identification information, if the content of the two area settings match, it may be impossible to distinguish between the first image data and the second image data. In contrast, by using the identification information, it is possible to reliably distinguish between the first image data and the second image data.

[0010] (Configuration 3) In configuration 1 or 2, the first request may include first identification information, and the first data may include second identification information corresponding to the first identification information. The second request may include third identification information different from the first identification information, and the second data may include fourth identification information corresponding to the third identification information and different from the second identification information. The device may place the second image at a second position in the composite image based on the fact that the received second data includes the fourth identification information.

[0011] According to configuration 3, the device can determine the position where the image should be output in the composite image by distinguishing at least the first image data and the second image data based on whether the first data received from the camera includes the second identification information and based on whether the second data received from the camera includes the fourth identification information. For example, if the content of the area setting is transmitted to the device instead of the identification information, it may be impossible to distinguish between the first image data and the second image data if the content of the two area settings match. In contrast, by using the second identification information and the fourth identification information, it is possible to reliably distinguish between the first image data and the second image data.

[0012] (Configuration 4) In configuration 3, the first identification information and the third identification information may be set regardless of which area of ​​the imaging area the image data corresponding to is to be transmitted to the camera.

[0013] According to the fourth aspect, by using identification information that is independent of the content of the area setting, it is possible to reduce the possibility of confusing the first image data and the second image data in the device.

[0014] (Configuration 5) In configuration 3 or 4, the first identification information and the third identification information may be either one of two values.

[0015] According to configuration 5, the first identification information and the third identification information can be expressed by data indicating two values ​​(for example, one bit), so that the amount of data to be transmitted can be reduced.

[0016] (Configuration 6) In any of configurations 2 to 5, the first identification information may be stored in a header defined in the UVC (USB Video Class) standard.

[0017] According to configuration 6, the identification information is stored in a header separate from the image data, so there is no effect on the image data. Also, since it is only necessary to process the header defined by the standard, the process for identification is stabilized.

[0018] (Configuration 7) In any of configurations 2 to 6, the first identification information may be stored in the first image data.

[0019] According to configuration 7, the first image data and the second image data can be distinguished from each other using the image data itself. Furthermore, since the identification information is stored in the image data, the amount of data required to transmit the identification information can be reduced.

[0020] (Configuration 8) In configuration 7, the first image data may be stored as data corresponding to an image of the edge of the first region.

[0021] According to configuration 8, it is possible to suppress degradation in the quality of the first image corresponding to the first image data as seen by the user.

[0022] (Configuration 9) In any of configurations 1 to 8, the first request may include a request to set image data included in the data to be transmitted to the camera to image data corresponding to a first region of the imaging region. The second request may include a request to set image data included in the data to be transmitted to the camera to image data corresponding to a second region of the imaging region. The camera may transmit data including the set image data in response to a third request, which is a request to transmit data including the image data from the camera.

[0023] According to configuration 9, by determining the timing of sending the third request depending on the timing of sending the first request and / or the second request, it is possible to optimize the reflection of the settings of the first area and / or the second area in the image data sent by the camera.

[0024] (Configuration 10) In any of configurations 1 to 9, the device may transmit a fourth request for setting an image size of image data to be transmitted from the camera before transmitting the first request and the second request. The first request and the second request may not include a request to change the image size.

[0025] According to configuration 10, the process of changing the image size in the camera may take a relatively long time, but since the first request and the second request do not include a request to change the image size, the first request and / or the second request can be reflected in the camera more quickly, thereby reducing the possibility of delays in data transmission from the camera to the device.

[0026] (Configuration 11) In configuration 10, the image size that is set may be smaller than the image size specified in the specifications.

[0027] According to configuration 11, processing such as optical zooming becomes possible.

[0028] (Configuration 12) According to another embodiment, there is provided a camera connected to a device via a wired or wireless connection. The device alternately transmits to the camera a first request for causing the camera to transmit image data corresponding to a first region of the camera's imaging area and a second request for causing the camera to transmit image data corresponding to a second region of the imaging area. In accordance with the first request, the camera generates first image data corresponding to the first region from overall image data corresponding to the imaging area and transmits the first data including the first image data to the device. In accordance with the second request, the camera generates second image data corresponding to the second region from the overall image data and transmits the second data including the second image data to the device. The first request includes first identification information, and the first data includes second identification information corresponding to the first identification information. Configuration 12 provides an effect similar to that of Configuration 1.

[0029] (Configuration 13) In configuration 12, the first identification information may be stored in a header defined in the UVC (USB Video Class) standard. According to configuration 13, the same effect as in configuration 6 can be obtained.

[0030] (Configuration 14) In configuration 12 or 13, the first identification information may be stored in the first image data. According to configuration 14, the same effect as in configuration 7 can be obtained.

[0031] (Configuration 15) According to yet another embodiment, there is provided a device connected to a camera by wire or wirelessly. The device alternately transmits to the camera a first request to cause the camera to transmit image data corresponding to a first region of the camera's imaging area and a second request to cause the camera to transmit image data corresponding to a second region of the camera's imaging area. In accordance with the first request, the camera generates first image data corresponding to the first region from overall image data corresponding to the imaging area and transmits the first data including the first image data to the device. In accordance with the second request, the camera generates second image data corresponding to the second region from the overall image data and transmits the second data including the second image data to the device. The device outputs a composite image including a first image based on the first image data included in the received first data and a second image based on the second image data included in the received second data. Configuration 15 provides an effect similar to that of Configuration 1.

[0032] (Configuration 16) In configuration 15, the first request may include first identification information, and the first data may include second identification information corresponding to the first identification information. The device may place the first image at a first position in the composite image based on the received first data including the second identification information, and may place the second image at a second position in the composite image based on the received second data not including the second identification information. Configuration 16 provides the same effect as configuration 2.

[0033] (Configuration 17) In configuration 15 or 16, the first request may include first identification information, and the first data may include second identification information corresponding to the first identification information. The second request may include third identification information different from the first identification information, and the second data may include fourth identification information corresponding to the third identification information and different from the second identification information. The device may place the second image at a second position in the composite image based on the fact that the received second data includes the fourth identification information. Configuration 17 provides an effect similar to that of configuration 3.

[0034] (Configuration 18) According to yet another embodiment, there is provided a method executed in a system including a device and a camera connected to the device via a wired or wireless connection. The method includes the steps of: the device alternately transmitting to the camera a first request to cause the camera to transmit image data corresponding to a first region of the camera's imaging area and a second request to cause the camera to transmit image data corresponding to a second region of the camera's imaging area; the camera generating first image data corresponding to the first region from overall image data corresponding to the imaging area in accordance with the first request and transmitting the first data including the first image data to the device; the camera generating second image data corresponding to the second region from the overall image data in accordance with the second request and transmitting the second data including the second image data to the device; and the device outputting a composite image including a first image based on the first image data included in the received first data and a second image based on the second image data included in the received second data. Configuration 18 provides an effect similar to that of Configuration 1. [Brief explanation of the drawings]

[0035] [Figure 1] FIG. 1 is a schematic diagram showing an example of the configuration of a system according to an embodiment of the present invention. [Figure 2] 10A to 10C are diagrams for illustrating an example of imaging processing and display processing in the system according to the present embodiment. [Figure 3] FIG. 10 is a diagram showing an example of a processing result in the system according to the present embodiment. [Figure 4] FIG. 10 is a diagram showing another example of a processing result in the system according to the present embodiment. [Figure 5] FIG. 10 is a diagram showing an example of communication processing between a device and a camera in the system according to the present embodiment. [Figure 6] FIG. 10 is a diagram showing another example of communication processing between a device and a camera in the system according to the present embodiment. [Figure 7] 10A and 10B are diagrams illustrating an example of embedding identification information into an image in the system according to the present embodiment. [Figure 8] 10 is a flowchart showing an example of a processing procedure in a device of the system according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0036] The present embodiment will be described in detail with reference to the drawings, in which the same or corresponding parts are designated by the same reference numerals and description thereof will not be repeated.

[0037] [A. System configuration example] First, an example of the configuration of a system 1 according to the present embodiment will be described.

[0038] FIG. 1 is a schematic diagram showing an example configuration of a system 1 according to the present embodiment. The system 1 includes a device 100 and a camera 200 connected to the device 100 via a wired or wireless connection. The camera 200 is capable of capturing moving images. The camera 200 may be detachable from the device 100. The camera 200 may be, for example, a camera for video calls or a surveillance camera. A plurality of cameras 200 may be connectable to the device 100. The camera 200 may be connectable only to a specific device, or may be connectable to various devices.

[0039] Device 100 is capable of displaying an image captured by camera 200. Device 100 includes, for example, a processor 102, memory 104, storage 110, an input unit 106, a display unit 108, and a communication unit 120. Device 100 may be, for example, a personal computer, a game console, a television, a smartphone, or the like.

[0040] The processor 102 is a processing entity (processing means) for executing processing in the device 100. The processor 102 includes, for example, a central processing unit (CPU) and a graphics processing unit (GPU). The processor 102 loads a program stored in the storage 110 into the memory 104 and executes the program.

[0041] The memory 104 is a volatile storage medium accessible by the processor 102, and includes, for example, a dynamic random access memory (DRAM) and a static random access memory (SRAM).

[0042] Storage 110 is a non-volatile storage medium accessible by processor 102, and includes, for example, flash memory and a hard disk. Storage 110 may also include, for example, a storage medium that is removable from device 100, such as a cartridge or optical disk.

[0043] The storage 110 stores, for example, a system program 112 and an application program 114 .

[0044] The system program 112 includes computer-readable instructions for basic processing such as communication processing between the device 100 and the camera 200, and processing image data received from the camera 200, etc.

[0045] The application programs 114 include computer-readable instructions for executing various applications.

[0046] As used herein, the term "processor" encompasses at least processing circuits that perform processing according to computer-readable instructions, such as CPUs and GPUs, and hardwired circuits, such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays).

[0047] As used herein, the term “memory” encompasses at least memory 104 and storage 110 .

[0048] Device 100 may employ a system on chip (SoC) that integrates functions such as a processor and memory. In this case, the processor and memory exist in the same integrated circuit. Therefore, in this specification, the terms "processor" and "memory" encompass both independent configurations and integrated configurations.

[0049] The input unit 106 accepts user operations. The input unit 106 includes, for example, a keyboard, a mouse, a game controller, buttons, a cross key, a microphone, etc. The input unit 106 may be an interface with a device that accepts user operations. In other words, the device that accepts user operations may be located outside the device 100.

[0050] Display unit 108 displays images or videos generated by processing by processor 102. Display unit 108 includes, for example, an LCD (Liquid Crystal Display) or an organic EL display. Display unit 108 may be an interface with a display device. In other words, the display device may be external to device 100.

[0051] The communication unit 120 transmits and receives data to and from the camera 200. The communication unit 120 may be connected to the camera 200 via a wired connection or wirelessly. In the case of a wired connection, for example, a USB (Universal Serial Bus) connection or a parallel connection may be used. In the case of a wireless connection, for example, Bluetooth (registered trademark), ZigBee (registered trademark), and wireless LAN (IEEE 802.11 standard) may be used.

[0052] The device 100 may be a general-purpose personal computer. In this case, the device 100 may have a driver responsible for communication with the camera 200, as described below, in addition to or instead of the system program 112.

[0053] The camera 200 includes, for example, an imaging element 210, a communication unit 220, and an image processing unit 230.

[0054] The imaging element 210 outputs image data (hereinafter also referred to as "whole image data") based on incident light. The imaging element 210 includes, for example, a CCD (Charge-Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. A lens may be disposed in front of the imaging element 210. The range (field of view) in which the imaging element 210 can generate image data is also referred to as the "imaging area."

[0055] The communication unit 220 transmits and receives data to and from the device 100. The communication unit 220 has the same configuration as the communication unit 120.

[0056] The image processing unit 230 processes the overall image data output from the imaging element 210 and transmits the processing result to the device 100 via the communication unit 220. The processing performed by the image processing unit 230 may be in accordance with a request and / or settings from the device 100. The image processing unit 230 may include a buffer 232 that temporarily stores the overall image data output from the imaging element 210.

[0057] The image processing unit 230 may include, for example, firmware 234. The firmware 234 includes computer-readable instructions for executing processing by the image processing unit 230. The image processing unit 230 may include a processor, an ASIC, and an FPGA, which are not shown.

[0058] [B. Imaging Processing and Display Processing] Next, imaging processing and display processing in system 1 according to the present embodiment will be described.

[0059] FIG. 2 is a diagram illustrating an example of imaging processing and display processing in system 1 according to the present embodiment. Referring to FIG. 2, image processing unit 230 of camera 200 generates first image data 262 corresponding to first region 252 of imaging region 250 of camera 200 and second image data 264 corresponding to second region 254 of imaging region 250 of camera 200 based on overall image data output by imaging element 210 (FIG. 1) in accordance with communication processing described below. For example, image processing unit 230 alternately generates first image data 262 and second image data 264. When image processing unit 230 generates images at 30 fps (frames per second), first image data 262 may be generated at 15 fps, and second image data 264 may be generated at 15 fps. First region 252 and second region 254 may be set as ROIs (Regions of Interest). When the device 100 and the camera 200 are connected via USB and data is exchanged in accordance with the UVC (USB Video Class) standard, which is the camera's communication specification, the ROI may also be set in accordance with the UVC standard.

[0060] The communication unit 220 of the camera 200 sequentially transmits the image data generated by the image processing unit 230 to the device 100.

[0061] Image data from camera 200 that is sequentially received by communication unit 120 of device 100 is stored in buffer 140. Buffer 140 may include areas for storing first image data 262 and second image data 264. The first image data 262 and second image data 264 are sequentially updated in each area, so that the latest first image data 262 and latest second image data 264 are stored in buffer 140.

[0062] The image synthesis unit 130 generates a synthesized image 170 including a first image 172 based on the first image data 262 stored in the buffer 140 and a second image 174 based on the second image data 264. The image synthesis unit 130 may also generate the synthesized image 170 based on an application image 160 generated by the application execution unit 150 that executes the application program 114 (FIG. 1). The generation cycle (update cycle) of the synthesized image 170 may be shorter than the cycle at which the image data is transmitted from the camera 200.

[0063] Image synthesis unit 130 outputs the generated synthetic image 170. The process of outputting synthetic image 170 includes at least some of the following processes: displaying synthetic image 170 on display unit 108 (FIG. 1), outputting a signal for displaying synthetic image 170 to an external display device or the like, transmitting data for displaying synthetic image 170 via a network or the like, and storing data for displaying synthetic image 170 in a storage medium.

[0064] FIG. 3 is a diagram showing an example of a processing result in system 1 according to the present embodiment. In the example shown in FIG. 3, two people are present in imaging area 250 of camera 200. For example, first area 252 is set around the face of one person, and second area 254 is set around the face of the other person. In this case, composite image 170 generated based on first image data 262 corresponding to first area 252 and second image data 264 corresponding to second area 254 includes images around the faces of each person. In composite image 170, the positions and sizes at which first image 172 and second image 174 are displayed may or may not be set in advance. As an example, the display position of at least one of first image 172 and second image 174 may be moved depending on the execution status of application program 114.

[0065] Each of the first region 252 and the second region 254 can be set arbitrarily in the imaging region 250. Therefore, it is also possible to set the entire imaging region 250 as the first region 252 and / or the second region 254.

[0066] Fig. 4 is a diagram showing another example of a processing result in system 1 according to the present embodiment. In the example shown in Fig. 4, a person is present in imaging area 250 of camera 200. For example, first area 252 is set around the person's face, and the entire imaging area 250 is set as second area 254. In this case, composite image 170 includes first image 172 showing the area around the person's face and second image 174 showing the entire field of view of camera 200.

[0067] The first region 252 and the second region 254 can be set in any manner.

[0068] Image synthesis unit 130 may process at least one of first image 172 and second image 174. In Fig. 4, second image 174 includes a frame object 176 that indicates an area corresponding to first image 172. That is, image synthesis unit 130 generates second image 174 by performing processing to superimpose frame object 176 on second image data 264 that corresponds to second area 254.

[0069] 4, by superimposing a frame object 176 indicating the area corresponding to the first image 172, the user can know at a glance which area of ​​the imaging area 250 of the camera 200 is zoomed in. In addition, the user can know at a glance how the zoomed area will be displayed.

[0070] [C. Communication Processing] Next, a communication process between device 100 and camera 200 in system 1 according to the present embodiment will be described.

[0071] The device 100 provides requests and / or settings to the camera 200 at the appropriate times to cause the camera 200 to transmit the first image data 262 and the second image data 264 .

[0072] FIG. 5 is a diagram showing an example of communication processing between device 100 and camera 200 in system 1 according to the present embodiment.

[0073] 5, device 100 transmits initial settings 180 (shown as "Set Format" in FIG. 5) required for transmitting image data to camera 200 (sequence SQ2). Initial settings 180 include settings for the data format and / or image size of the image data to be transmitted by camera 200. Camera 200 (image processing unit 230) sets a default image size in accordance with initial settings 180.

[0074] If an aspect ratio (e.g., 16:9, 3:4, etc.) is preset in camera 200, initial setting 180 may include only the setting of the number of pixels on the short side (e.g., 360 pixels, 180 pixels, etc.). When the aspect ratio is set to 16:9 and 360 pixels is set, camera 200 sets 640 pixels x 360 pixels as the image size. The image size set by initial setting 180 may be smaller than the image size of the entire image data output by imaging element 210 (e.g., 1920 pixels x 1080 pixels). In other words, the image size set by initial setting 180 may be smaller than the image size specified in the specifications.

[0075] The frequency (fps) at which the camera 200 transmits image data may be set in advance or may be included in the initial settings 180 .

[0076] Next, device 100 transmits first region setting 182 (shown as "Set ROI1" in FIG. 5) for setting first region 252 as a region for which image data should be generated to camera 200 (sequence SQ4). Camera 200 (image processing unit 230) sets first region 252 as an ROI in accordance with first region setting 182.

[0077] The first region setting 182 includes a request to set the image data included in the data to be transmitted by the camera 200 to the first image data 262 corresponding to the first region 252 of the imaging region 250.

[0078] Next, the camera 200 starts generating image data in accordance with the set ROI in synchronization with the transmission period (sequence SQ6). When the first region 252 is set as the ROI, the camera 200 (image processing unit 230) generates first image data 262 corresponding to the first region 252. The camera 200 (image processing unit 230) transmits data including the generated first image data 262 to the device 100 (sequence SQ8).

[0079] Device 100 transmits second region setting 184 (shown as "Set ROI2" in FIG. 5) for setting second region 254 as a region for which image data should be generated to camera 200 (sequence SQ10). Camera 200 (image processing unit 230) sets second region 254 as an ROI in accordance with second region setting 184.

[0080] The second region setting 184 includes a request to set the image data included in the data to be transmitted by the camera 200 to the second image data 264 corresponding to the second region 254 of the imaging region 250.

[0081] Next, the camera 200 starts generating image data in accordance with the set ROI in synchronization with the transmission period (sequence SQ12). If the second region 254 is set as the ROI, the camera 200 (image processing unit 230) generates second image data 264 corresponding to the second region 254. The camera 200 (image processing unit 230) transmits data including the generated second image data 264 to the device 100 (sequence SQ14).

[0082] Thereafter, the same processing as in the sequences SQ4 to SQ14 is repeated.

[0083] As shown in FIG. 5, by sequentially providing the camera 200 with a first area setting 182 and / or a second area setting 184 (hereinafter sometimes collectively referred to as "area settings") for the device 100 to generate image data, the device 100 can acquire image data corresponding to different areas of the imaging area 250.

[0084] In this way, device 100 alternately switches between a first area setting 182 (corresponding to a first request) for causing camera 200 to transmit image data corresponding to a first area 252 of imaging area 250 of camera 200, and a second area setting 184 (corresponding to a second request) for causing camera 200 to transmit image data corresponding to a second area 254 of imaging area 250, and transmits the image data to camera 200. Note that, as will be described later, first area setting 182 and second area setting 184 do not need to be switched strictly alternately.

[0085] Camera 200 generates first image data 262 corresponding to first region 252 from overall image data (image data output by image sensor 210 shown in FIG. 1 ) corresponding to imaging region 250 in accordance with first region setting 182, and transmits data including first image data 262 to device 100. Camera 200 also generates second image data 264 corresponding to second region 254 from the overall image data in accordance with second region setting 184, and transmits data including second image data 264 to device 100.

[0086] In addition, in sequences SQ6 and SQ12, camera 200 may start generating image data according to the set ROI based on an image transmission request from device 100. In this case, device 100 sequentially transmits image transmission requests to camera 200 in synchronization with the transmission period. That is, camera 200 may transmit data including the set image data in response to an image transmission request (third request) that requests camera 200 to transmit data including image data.

[0087] Therefore, the "first request" may include a first region setting 182 and an image transmission request requesting image data corresponding to the first region setting 182. Similarly, the "second request" may include a second region setting 184 and an image transmission request requesting image data corresponding to the second region setting 184.

[0088] As shown in FIG. 5, before transmitting the first area setting 182 and the second area setting 184, the device 100 transmits to the camera 200 an initial setting 180 (corresponding to a fourth request) for setting the image size of the image data to be transmitted by the camera 200.

[0089] Fig. 6 is a diagram showing another example of communication processing between device 100 and camera 200 in system 1 according to the present embodiment. Fig. 6 shows an example in which device 100 transmits second region setting 184 to camera 200, and then transmits first region setting 182 to camera 200 after a time exceeding one transmission cycle has elapsed.

[0090] 6, after camera 200 sets second region 254 as the ROI in accordance with second region setting 184, camera 200 maintains the previously set ROI until a new region setting is received. Therefore, camera 200 transmits second image data 264 twice in succession. That is, camera 200 may transmit either image data in succession while alternately transmitting first image data 262 and second image data 264.

[0091] In this way, device 100 only needs to acquire the first image data 262 and the second image data 264 within a period that allows the user to recognize that the first image 172 and the second image 174 were captured at approximately the same time, and the transmission of the first image data 262 and the second image data 264 does not have to be strictly alternated. In other words, the transmission order and transmission pattern when the first image data 262 and the second image data 264 are alternately transmitted may be set and changed arbitrarily for the convenience of device 100. For example, the transmission frequency of one of the first image data 262 and the second image data 264 may be higher.

[0092] [D. Area settings] Next, the area setting transmitted from the device 100 to the camera 200 will be described.

[0093] The first region setting 182 and the second region setting 184 may include identification information for identifying each region setting. For example, the first region setting 182 may include "0" as the identification information, and the second region setting 184 may include "1" as the identification information. Alternatively, when there are two types of region settings, the identification information may be assigned to only one of the region settings. For example, the first region setting 182 may include the identification information, and the second region setting 184 may not include the identification information.

[0094] In this way, the first region setting 182 (corresponding to the first request) may include first identification information, and the second region setting 184 (corresponding to the second request) may include third identification information different from the first identification information. The identification information included in the first region setting 182 (corresponding to the first identification information) and the identification information included in the second region setting 184 (corresponding to the third identification information) may be either one of two values.

[0095] It should be noted that there is no need to associate the setting contents of first region setting 182 and second region setting 184 with the values ​​of the identification information included in first region setting 182 and second region setting 184. The identification information is used to identify which of the multiple region settings the region setting is, and it is not necessary to identify the content of the region setting. In other words, the identification information included in first region setting 182 and the identification information included in second region setting 184 may be set regardless of which region of imaging region 250 corresponds to image data that camera 200 is to transmit.

[0096] Furthermore, as will be described later, the camera 200 can determine the identification information to be added to the generated image data based on the identification information included in the area setting (and / or the absence of identification information).

[0097] By employing such identification information, even if the setting contents of the first region setting 182 and the second region setting 184 become identical, for example, the device 100 can determine which region setting the image data corresponds to. As an example, consider a case where the first region 252 and the second region 254 change relatively and the first region 252 and the second region 254 end up matching. In this case, the device 100 may not be able to determine which of the first image 172 and the second image 174 in the composite image 170 should be updated based on the image data transmitted from the camera 200. However, if the identification information is employed as described above, even if the first region 252 and the second region 254 end up matching, it is clear which of the first image 172 and the second image 174 should be updated based on the image data transmitted from the camera 200. Note that in other embodiments, the identification information need not be used. As an example, if the first area 252 and the second area 254 are the same, both the first image 172 and the second image 174, i.e., the first image data 262 and the latest second image data 264 stored in the buffer 140, may be updated based on the received image data.

[0098] The first region setting 182 and the second region setting 184 (region setting) may include, as parameters for setting the ROI, the coordinates (X and Y coordinates) of the reference position of the ROI (for example, the upper left vertex) and the number of pixels on one side (for example, the long side) of the ROI. If the aspect ratio is set in advance, the image size of the ROI is calculated by setting the number of pixels on one side of the ROI.

[0099] For example, if (100, 100, 480) is set as the parameters (X coordinate, Y coordinate, number of pixels on the long side) of the first region setting 182 or the second region setting 184, and the aspect ratio is 16:9, the image size is calculated to be 480 pixels x 270 pixels. As a result, a rectangular region with vertices at coordinates (100, 100), (100, 370), (580, 100), and (580, 370) is set as the ROI.

[0100] Image processing unit 230 of camera 200 extracts image data corresponding to the region set as ROI from the overall image data output from image sensor 210. Note that image sensor 210 may directly output image data corresponding to the region set as ROI, rather than temporarily storing the overall image data output from image sensor 210 and then extracting it.

[0101] In the above example, the extracted image data is 480 pixels x 270 pixels, so the image processor 230 of the camera 200 resizes (reduces or enlarges) the extracted image so that it matches the image size (e.g., 640 pixels x 360 pixels) set by the initial setting 180. The image processor 230 of the camera 200 may upscale (increase the resolution of) the extracted image. If the image size of the ROI matches the image size set by the initial setting 180, the resizing process may be omitted.

[0102] The image size of the image data generated by the camera 200 is set in advance by the initial setting 180. In addition, the image processing unit 230 of the camera 200 resizes the image. Therefore, the first region setting 182 or the second region setting 184 does not need to include a request to change the image size.

[0103] As described above, in system 1 according to the present embodiment, image data required for display on device 100 is generated in camera 200, eliminating the need to transmit unnecessary image data and enabling effective use of the communication bandwidth between device 100 and camera 200. Furthermore, regardless of the position and size of the set ROI, image data of a preset image size is transmitted, thereby suppressing fluctuations in the processing load for display on device 100.

[0104] [E. Image Data Identification] Next, an example of a method for identifying to which ROI image data received by device 100 from camera 200 corresponds will be described.

[0105] (e1: storing identification information in the header) Identification information for identifying the corresponding ROI may be included in the data transmitted from the camera 200 to the device 100 .

[0106] Specifically, the first data transmitted from camera 200 and including first image data 262 may include identification information (second identification information) corresponding to the identification information (first identification information) of first area setting 182. Furthermore, the second data transmitted from camera 200 and including second image data 264 may include identification information (fourth identification information) corresponding to the identification information (third identification information) of second area setting 184. If second area setting 184 does not include identification information, the second data transmitted from camera 200 and including second image data 264 may also not include identification information. The first identification information and the second identification information, and the third identification information and the fourth identification information may be the same or different.

[0107] The identification information included in the data may be in any format, for example, the identification information for identifying the ROI may be stored in a header defined in the UVC standard.

[0108] The camera 200 may include, in the header, the identification information (for example, "0" or "1") of the area setting corresponding to the generated image data.

[0109] If there is no identification information for the region setting corresponding to the generated image data, the identification information may not be included in the header. In that case, if there is identification information for the region setting corresponding to the generated image data, information indicating the presence of the setting information may be included in the header. For example, if a partial region and the entire region of the imaging region 250 are set as ROIs, identification information indicating that the region has been set (or identification information the same as the identification information for the region setting) may be added to the image data representing the partial region, and no identification information may be added to the image data representing the entire region.

[0110] Alternatively, the camera 200 may include in the header a value (for example, a character string such as "ROI1" or "ROI2") calculated according to a predetermined rule from the identification information of the region setting corresponding to the generated image data.

[0111] The identification information stored in the header may be written in key-value format.

[0112] As described above, device 100 can determine to which region setting the received image data corresponds, based on the identification information included in the data received from camera 200. By employing such a configuration, the process for determining the ROI corresponding to the image data in device 100 can be stabilized.

[0113] (e2: Embedding identification information into images) Identification information for identifying which area setting the image is based on may be embedded in the image, i.e., the identification information for identifying the corresponding area setting may be stored in the image data.

[0114] After generating image data, the camera 200 rewrites specific pixel values ​​of the generated image data based on the area setting corresponding to the generated image data.

[0115] 7A and 7B are diagrams illustrating an example of embedding identification information into an image in system 1 according to the present embodiment. As an example, Fig. 7A and Fig. 7B show a luminance (Y) map of image data in the YCbCr format.

[0116] The identification information may be stored in the image data corresponding to the edge image of the ROI. For example, the pixel value (brightness value) of a specific coordinate of the image data (the upper left vertex in the example shown in FIG. 7) may be forcibly set to "0" (FIG. 7(A)) or "1" (FIG. 7(B)). "0" may be interpreted as meaning that the image was generated according to the first region setting 182, and "1" may be interpreted as meaning that the image was generated according to the second region setting 184.

[0117] When device 100 receives image data with embedded identification information from camera 200, it determines which area setting the received image data is based on, based on pixel values ​​at specific coordinates.

[0118] Although the pixel values ​​at specific coordinates in the received image data are different from the original values, if the image size of the image data is sufficiently large, the visual discomfort can be ignored, and the image data may be used as is to generate the composite image 170. Alternatively, the device 100 may estimate and correct the pixel value at the specific coordinates from the pixel values ​​around the specific coordinates.

[0119] The identification information can be embedded in any pixel position. The identification information can be embedded in the image in the same way even when image data is generated in any format, such as RGB or CMYK, instead of YCbCr.

[0120] By employing a method of embedding identification information in an image, device 100 can determine which area setting the received image data corresponds to, without storing identification information in a header or the like.

[0121] (e3: Timing-based identification) As described above, device 100 provides a region setting to camera 200, whereby an ROI is set in camera 200, and image data corresponding to the set ROI is transmitted from camera 200. Therefore, when device 100 receives image data from camera 200, it may identify the latest region setting in the past, more than a predetermined delay time from the timing of receiving the image data, and determine that the image data corresponds to the identified region setting.

[0122] The specified delay time may be determined taking into consideration the time required to set the ROI in the camera 200, the time required to generate image data corresponding to the ROI in the camera 200, and the time required to transmit the image data from the camera 200 to the device 100.

[0123] [F. Processing in Device 100] Next, an example of processing in the device 100 will be described.

[0124] Fig. 8 is a flowchart showing an example of a processing procedure in device 100 of system 1 according to the present embodiment. The steps shown in Fig. 8(A) and Fig. 8(B) may be implemented by processor 102 of device 100 executing system program 112 and / or application program 114. Fig. 8(A) shows processing related to a request from device 100 to camera 200. Fig. 8(B) shows processing related to data reception in device 100 from camera 200.

[0125] 8(A), when an image display request is generated (YES in step S100), device 100 transmits initial setting 180 to camera 200 (step S102). The image display request may be issued by, for example, a user operation or may be issued by application program 114 (application execution unit 150 shown in FIG. 2).

[0126] Next, the device 100 transmits the first region setting 182 to the camera 200 in synchronization with the transmission cycle (step S104). Next, the device 100 transmits the second region setting 184 to the camera 200 in synchronization with the transmission cycle (step S106).

[0127] The device 100 repeats the processes of steps S104 and S106 until an instruction to stop image display is received.

[0128] When an instruction to stop image display is given (YES in step S108), the device 100 ends the processing. The device 100 may end the processing after transmitting a request to reset the settings to the camera 200.

[0129] After steps S104 and S106, the device 100 may transmit an image transmission request to the camera 200.

[0130] 8(B), when an image display request occurs (YES in step S100), device 100 determines whether data has been received from camera 200 (step S120). When data has been received from camera 200 (YES in step S120), device 100 determines whether the received data includes first image data 262 or second image data 264 (step S122), and stores the image data included in the received data in the corresponding area of ​​buffer 140 (step S124).

[0131] In step S122, it is determined which area setting the image data corresponds to by one of the methods described above.

[0132] Next, device 100 generates composite image 170 based on the first image data 262 and second image data 264 stored in buffer 140 (step S126). Then, device 100 outputs the generated composite image 170 to display unit 108 (step S128). Note that if only one of the first image data 262 and the second image data 264 is stored in buffer 140, the processes of steps S126 and S128 may be skipped.

[0133] The device 100 repeats the processes of steps S120 to S128 until an instruction to stop image display is received.

[0134] When an instruction to stop image display is given (YES in step S130), the device 100 ends the process.

[0135] The processes shown in FIGS. 8(A) and 8(B) may be executed in parallel or serially.

[0136] Device 100 places first image 172 based on first image data 262 and second image 174 based on second image data 264 at predetermined positions in composite image 170 (see FIGS. 3 and 4). Device 100 places first image 172 based on first image data 262 at a predetermined position (first position: for example, the right side) in composite image 170 based on the first data (data including first image data 262) received from camera 200, based on the inclusion of identification information indicating first region setting 182. Similarly, device 100 places second image 174 based on second image data 264 at another predetermined position (second position: for example, the left side) in composite image 170 based on the second data (data including second image data 264) received from camera 200, based on the inclusion of identification information indicating second region setting 184.

[0137] Alternatively, if the second region setting 184 does not include identification information, the device 100 may place the second image 174 based on the second image data 264 at another predetermined position (second position: for example, the left side) in the composite image 170 based on the second data (data including the second image data 264) received from the camera 200 not including identification information indicating the second region setting 184.

[0138] [G. Variations] In the above embodiment, an example was shown in which two regions for which image data should be generated are prepared, but three or more regions may be set by alternating between them. In this case, the composite image 170 will include three or more images. Note that in this case, too, it can be said that requests to transmit image data corresponding to two of the three or more regions to the camera are alternately switched.

[0139] The region for which image data should be generated may be determined by any method. For example, it may be set in advance, may be set by the user, or an ROI may be set around a person's face based on the results of face recognition. In the case of determination by face recognition, face recognition is performed sequentially, so that even if the region in the image capture area 250 where a person exists changes over time, the ROI can be tracked around the person's face. Such face recognition processing may be performed by either the device 100 or the camera 200. As an example, the device 100 may identify the position of the person's face by image processing the received image data and generate a region setting corresponding to the identified position. On the other hand, when face recognition processing is performed in the camera 200, the device 100 may transmit a request necessary for face authentication (e.g., a request to recognize the face of a specific person present in the image capture area 250) to the camera 200. When information on the range of the ROI set in the device 100 is required, the camera 200 may transmit the range (vertex coordinates) of the ROI corresponding to the image data to the device 100 in addition to the image data to be generated.

[0140] 3 and 4 illustrate an example in which device 100 displays images based on image data corresponding to multiple regions transmitted from camera 200. However, multiple images based on multiple image data do not necessarily need to be displayed at all times. When device 100 does not display multiple images based on image data corresponding to multiple regions (i.e., when displaying only images based on image data corresponding to a specific region), the image data used for display may be captured more frequently than when multiple image data corresponding to multiple regions are captured. For example, device 100 may sometimes display only an image showing the entire imaging region 250 (e.g., only second image 174 shown in FIG. 4 is displayed). In this case, an ROI does not need to be set. When zooming in on a specific region, composite image 170 including first image 172 and second image 174 shown in FIG. 4 may be displayed. Furthermore, when the region to be zoomed (zoom region) is determined, only first image 172 shown in FIG. 4 may be displayed.

[0141] The data transmitted from camera 200 may include the generated image data as is (i.e., in an uncompressed state), or may include compressed image data. When image data is transmitted uncompressed, compression processing in camera 200 and decompression processing in device 100 are not required, so an increase in processing load can be suppressed. When image data is transmitted in a compressed form, more separate data can be transmitted.

[0142] Camera 200 may be a general-purpose camera. However, the general-purpose camera may not be able to transmit data including identification information. In that case, the general-purpose camera may transmit data including an ROI to device 100. Device 100 may determine an ROI corresponding to the received image data according to the value of the ROI included in the received data, and generate composite image 170 according to the determined ROI.

[0143] The ROI may be set in a manner conforming to the UVC standard as described above, or in a manner conforming to any communication standard.

[0144] As an example of processing at least one of the first image 172 and the second image 174, the device 100 may generate an avatar based on at least one of the first image data 262 and the second image data 264. For example, the device 100 may generate an avatar that moves in accordance with the movement of the subject (user) in the first image data 262 or the second image data 264, or may generate an avatar whose facial expression changes in accordance with the facial expression of the subject (user). The device 100 can transmit data efficiently by setting an area necessary for control to generate the avatar.

[0145] Although device 100 outputs the generated composite image 170 to display unit 108, it may also output it to another output destination. For example, device 100 may transmit composite image 170 to another device 100 by outputting it to a transmitter, and the other device 100 may display composite image 170 or an image obtained by processing composite image 170. Note that device 100 may transmit at least one of first image data 262 and second image data 264, or image data based on these image data, to the other device 100. The other device 100 may generate and display an image using this image data or other image data.

[0146] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0147] 1 system, 100 device, 102 processor, 104 memory, 106 input unit, 108 display unit, 110 storage, 112 system program, 114 application program, 120, 220 communication unit, 130 image synthesis unit, 140, 232 buffer, 150 application execution unit, 160 application image, 170 synthesis image, 172 first image, 174 second image, 180 initial setting, 182 first area setting, 184 second area setting, 200 camera, 210 image sensor, 230 image processing unit, 234 firmware, 250 imaging area, 252 first area, 254 second area, 262 first image data, 264 second image data.

Claims

1. 1. A system comprising: a device; a camera connected to the device by wire or wirelessly; the device alternately switches between transmitting to the camera a first request for causing the camera to transmit image data corresponding to a first region of an imaging region of the camera and a second request for causing the camera to transmit image data corresponding to a second region of the imaging region; The camera is generating first image data corresponding to the first region from overall image data corresponding to the imaging region in accordance with the first request, and transmitting first data including the first image data to the device; generating second image data corresponding to the second region from the entire image data in accordance with the second request, and transmitting second data including the second image data to the device; The device outputs a composite image including a first image based on the first image data included in the received first data and a second image based on the second image data included in the received second data.

2. the first request includes first identification information, and the first data includes second identification information corresponding to the first identification information; The device comprises: placing the first image at a first position in the composite image based on the received first data including the second identification information; The system of claim 1 , further comprising: placing the second image at a second location in the composite image based on the received second data not including the second identification information.

3. the first request includes first identification information, and the first data includes second identification information corresponding to the first identification information; the second request includes third identification information different from the first identification information, and the second data includes fourth identification information corresponding to the third identification information and different from the second identification information; The system of claim 1 , wherein the device places the second image at a second position in the composite image based on the received second data including the fourth identification information.

4. The system according to claim 3 , wherein the first identification information and the third identification information are set regardless of which of the imaging areas the image data corresponding to is to be transmitted from the camera.

5. The system of claim 3 , wherein the first identification information and the third identification information are either one of two values.

6. 6. The system according to claim 2, wherein the first identification information is stored in a header defined in a USB Video Class (UVC) standard.

7. The system of any one of claims 2 to 5, wherein the first identification information is stored within the first image data.

8. The system according to claim 7 , wherein the first image data is stored in data corresponding to an image of a border in the first region.

9. the first request includes a request to set image data included in the data to be transmitted to the camera to image data corresponding to the first area of ​​the imaging area, the second request includes a request to set image data included in the data to be transmitted to the camera to image data corresponding to the second area of ​​the imaging area, The system described in any one of claims 1 to 5, wherein the camera transmits data including the set image data in response to a third request, which is a request to have the camera transmit data including image data.

10. the device transmits a fourth request for setting an image size of image data to be transmitted to the camera before transmitting the first request and the second request; The system according to any one of claims 1 to 5, wherein the first request and the second request do not include a request to change the image size.

11. The system according to claim 10 , wherein the set image size is smaller than the image size specified in the specifications.

12. A camera that is connected to a device by wire or wirelessly, the device alternately switches between transmitting to the camera a first request for causing the camera to transmit image data corresponding to a first region of an imaging region of the camera and a second request for causing the camera to transmit image data corresponding to a second region of the imaging region; The camera is generating first image data corresponding to the first region from overall image data corresponding to the imaging region in accordance with the first request, and transmitting first data including the first image data to the device; generating second image data corresponding to the second region from the entire image data in accordance with the second request, and transmitting second data including the second image data to the device; The first request includes first identification information, and the first data includes second identification information corresponding to the first identification information.

13. The camera according to claim 12 , wherein the first identification information is stored in a header defined in a USB Video Class (UVC) standard.

14. The camera of claim 12 , wherein the first identification information is stored within the first image data.

15. A device connected to a camera by wire or wirelessly, alternately transmitting to the camera a first request for causing the camera to transmit image data corresponding to a first region of an imaging region of the camera and a second request for causing the camera to transmit image data corresponding to a second region of the imaging region; The camera is generating first image data corresponding to the first region from overall image data corresponding to the imaging region in accordance with the first request, and transmitting first data including the first image data to the device; generating second image data corresponding to the second region from the entire image data in accordance with the second request, and transmitting second data including the second image data to the device; The device outputs a composite image including a first image based on the first image data included in the received first data and a second image based on the second image data included in the received second data.

16. the first request includes first identification information, and the first data includes second identification information corresponding to the first identification information; The device comprises: placing the first image at a first position in the composite image based on the received first data including the second identification information; The device of claim 15 , further comprising: placing the second image at a second location in the composite image based on the received second data not including the second identification information.

17. the first request includes first identification information, and the first data includes second identification information corresponding to the first identification information; the second request includes third identification information different from the first identification information, and the second data includes fourth identification information corresponding to the third identification information and different from the second identification information; The device of claim 15 , wherein the device places the second image at a second position in the composite image based on the received second data including the fourth identification information.

18. 1. A method carried out in a system including a device and a camera connected to the device via a wired or wireless connection, comprising: a step in which the device alternately switches between transmitting to the camera a first request to cause the camera to transmit image data corresponding to a first region of an imaging region of the camera and a second request to cause the camera to transmit image data corresponding to a second region of the imaging region; generating first image data corresponding to the first area from overall image data corresponding to the imaging area in accordance with the first request, and transmitting first data including the first image data to the device; generating second image data corresponding to the second region from the overall image data in accordance with the second request, and transmitting second data including the second image data to the device by the camera; The method comprises a step in which the device outputs a composite image including a first image based on the first image data included in the received first data and a second image based on the second image data included in the received second data.

Images