System, camera, apparatus and method

Abstract

The invention relates to a system, a camera, an apparatus and a method. The system includes a device and a camera connected to the device in a wired or wireless manner. The apparatus alternately switches and transmits a first request for causing the camera to transmit image data corresponding to the first region and a second request for causing the camera to transmit image data corresponding to the second region to the camera. The camera generates first image data corresponding to the first area according to the overall image data in accordance with the first request and transmits first data including the first image data to the device, and the camera generates second image data corresponding to the second area according to the overall image data in accordance with the second request and transmits the second image data to the device. And transmit second data including the second image data to the device. The device outputs a composite image including a first image based on the first image data and a second image based on the second image data.

Description

Technical Field

[0001] The present disclosure relates to a system, a camera, an apparatus and a method. Background Art

[0002] For example, Japanese Patent Application Laid-Open No. 2012-84968 discloses a configuration in which image data is transmitted from a camera body to a monitor device. Summary of the Invention

[0003] There is a demand for simultaneously outputting images of different regions in an image captured by a single camera. The present disclosure provides a new solution to this demand.

[0004] (Structure 1) According to a certain embodiment, a system includes a device and a camera connected to the device in a wired or wireless manner. The device alternately switches and sends a first request and a second request to the camera, wherein the first request is used to cause the camera to send image data corresponding to a first area in the camera's imaging area, and the second request is used to cause the camera to send image data corresponding to a second area in the imaging area. In accordance with the first request, the camera generates first image data corresponding to the first area based on the overall image data corresponding to the imaging area, and sends first data containing the first image data to the device. In accordance with the second request, the camera generates second image data corresponding to the second area based on the overall image data, and sends second data containing the second image data to the device. The device outputs a composite image containing the first image and the second image, wherein the first image is an image based on the first image data contained in the received first data, and the second image is an image based on the second image data contained in the received second data.

[0005] According to Configuration 1, the camera transmits first image data corresponding to the first area based on the first request and second image data corresponding to the second area based on the second request to the device. The data transmitted from the camera to the device need not include image data unnecessary for outputting a composite image, thereby enabling output of a composite image including images corresponding to at least two areas while reducing the amount of transmitted data.

[0006] For example, consider the process of transmitting entire image data from a camera to a device, and extracting image data required for a composite image from the entire image data. Information about areas of the entire image data that are not extracted is unnecessary. Transmitting the entire image data, which also includes this potentially unnecessary information, wastes the communication bandwidth between device 100 and camera 200. Configuration 1 reduces this waste of communication bandwidth.

[0007] (Structure 2) In Structure 1, the first request may include first identification information, and the first data may include second identification information corresponding to the first identification information. Alternatively, the device may place the first image at a first position in the composite image based on the fact that the received first data includes the second identification information, and place the second image at a second position in the composite image based on the fact that the received second data does not include the second identification information.

[0008] According to Configuration 2, the device can distinguish at least the first image data from the second image data based on whether the first data received from the camera contains the second identification information, thereby determining the position of the image to be output in the composite image. For example, if the area settings are sent to the device instead of the identification information, the first image data and the second image data may not be distinguished if the two area settings match. In contrast, using the identification information allows for reliable distinction between the first and second image data.

[0009] (Structure 3) In Structure 1 or 2, the first request may include first identification information, and the first data may include second identification information corresponding to the first identification information. Alternatively, the second request may include third identification information different from the first identification information, and the second data may include fourth identification information corresponding to the third identification information and different from the second identification information. Alternatively, the device may place the second image at the second position in the composite image based on the received second data including the fourth identification information.

[0010] According to Configuration 3, the device can distinguish at least the first image data from the second image data based on whether the first data received from the camera contains the second identification information and whether the second data received from the camera contains the fourth identification information, thereby determining the position of the image to be output in the composite image. For example, if the content of the area settings is sent to the device instead of the identification information, and the two area settings match, it may be impossible to distinguish the first image data from the second image data. In contrast, by using the second identification information and the fourth identification information, the first image data and the second image data can be reliably distinguished.

[0011] (Configuration 4) In Configuration 3, the first identification information and the third identification information may be set regardless of which area in the imaging area the camera is caused to transmit image data corresponding to.

[0012] According to Configuration 4, by using identification information that is independent of the content of the area setting, it is possible to reduce the possibility of mixing the first image data and the second image data in the device.

[0013] (Structure 5) In Structure 3 or 4, the first identification information and the third identification information may be either one of two values.

[0014] According to Configuration 5, the first identification information and the third identification information can be expressed using data representing a binary value (for example, 1 bit), and thus the amount of data to be transmitted can be reduced.

[0015] (Structure 6) In any of Structures 2 to 5, the first identification information may be stored in a header defined in the UVC (USB Video Class) standard.

[0016] According to Configuration 6, the identification information is stored in a header other than the image data, so that the image data is not affected. In addition, only the header, which is given a meaning by a standard, needs to be processed, so the process for discrimination is stabilized.

[0017] (Configuration 7) In any one of Configurations 2 to 6, the first identification information may be stored in the first image data.

[0018] According to Configuration 7, the first image data and the second image data can be distinguished using the image data itself. In addition, since the identification information is stored in the image data, the amount of data required to transmit the identification information can be reduced.

[0019] (Configuration 8) In Configuration 7, the first identification information may be stored in data corresponding to an image of an edge in the first region in the first image data.

[0020] According to Configuration 8, it is possible to suppress degradation in the quality of the first image corresponding to the first image data viewed by the user.

[0021] (Configuration 9) In any of Configurations 1 to 8, the first request may include a request for setting image data included in data to be transmitted by the camera to image data corresponding to a first area in the imaging area. The second request may include a request for setting image data included in data to be transmitted by the camera to image data corresponding to a second area in the imaging area. The camera may transmit data including the set image data in response to a third request that is a request for the camera to transmit data including image data.

[0022] According to Configuration 9, by determining the timing of sending the third request based on the timing of sending the first request and / or the second request, it is possible to optimize the reflection of the settings of the first area and / or the second area in the image data sent by the camera.

[0023] (Configuration 10) In any of Configurations 1 to 9, the device may transmit a fourth request for setting the image size of image data to be transmitted by the camera before transmitting the first and second requests. The first and second requests may not include a request to change the image size.

[0024] According to Configuration 10, while the image size change processing in the camera may take a relatively long time, the first and second requests do not include a request to change the image size. This allows the camera to process the first and / or second requests more quickly. As a result, the possibility of delays in data transmission from the camera to the device can be reduced.

[0025] (Configuration 11) In Configuration 10, the set image size may be smaller than the standard image size.

[0026] According to Configuration 11, processing such as optical zoom can also be performed.

[0027] (Configuration 12) According to another embodiment, a camera connected to a device via a wired or wireless connection is provided. The camera generates first image data corresponding to a first area based on overall image data corresponding to a camera area in response to a first request. In response to a second request, the camera generates second image data corresponding to a second area based on the overall image data. The camera then transmits first data containing the first image data to the device and second data containing the second image data to the device. The first request includes first identification information, and the first data includes second identification information corresponding to the first identification information. Configuration 12 achieves the same effects as Configuration 1.

[0028] (Configuration 13) In Configuration 12, the first identification information may be stored in a header defined in the UVC (USB Video Class) standard. According to Configuration 13, the same effects as those of Configuration 6 are obtained.

[0029] (Configuration 14) In Configuration 12 or 13, the first identification information may be stored in the first image data. According to Configuration 14, the same effects as those of Configuration 7 are obtained.

[0030] (Structure 15) According to another embodiment, a device connected to a camera in a wired or wireless manner is provided. The device alternately switches and sends a first request and a second request to the camera, wherein the first request is used to cause the camera to send image data corresponding to a first area in the camera's imaging area, and the second request is used to cause the camera to send image data corresponding to a second area in the imaging area. In accordance with the first request, the camera generates first image data corresponding to the first area based on the overall image data corresponding to the imaging area, and sends first data containing the first image data to the device. In accordance with the second request, the camera generates second image data corresponding to the second area based on the overall image data, and sends second data containing the second image data to the device. The device outputs a composite image containing the first image and the second image, wherein the first image is an image based on the first image data contained in the received first data, and the second image is an image based on the second image data contained in the received second data. According to Structure 15, the same effect as Structure 1 is obtained.

[0031] (Configuration 16) In Configuration 15, the first request may include first identification information, and the first data may include second identification information corresponding to the first identification information. Alternatively, the device may place the first image at a first position in the composite image based on the fact that the received first data includes the second identification information, and place the second image at a second position in the composite image based on the fact that the received second data does not include the second identification information. Configuration 16 provides the same effects as Configuration 2.

[0032] (Structure 17) In Structure 15 or 16, the first request may include first identification information, and the first data may include second identification information corresponding to the first identification information. Alternatively, the second request may include third identification information different from the first identification information, and the second data may include fourth identification information corresponding to the third identification information and different from the second identification information. Alternatively, the device may arrange the second image at the second position in the composite image based on the fact that the received second data includes the fourth identification information. With Structure 17, the same effects as those of Structure 3 are achieved.

[0033] (Structure 18) According to another embodiment, a method is provided, which is performed in a system including a device and a camera connected to the device by wire or wirelessly. The method includes the following steps: the device alternately switches and sends a first request and a second request to the camera, wherein the first request is used to cause the camera to send image data corresponding to a first area in the camera's imaging area, and the second request is used to cause the camera to send image data corresponding to a second area in the imaging area; the camera generates first image data corresponding to the first area based on the overall image data corresponding to the imaging area in accordance with the first request, and sends first data including the first image data to the device; the camera generates second image data corresponding to the second area based on the overall image data in accordance with the second request, and sends second data including the second image data to the device; and the device outputs a composite image including the first image and the second image, wherein the first image is an image based on the first image data included in the received first data, and the second image is an image based on the second image data included in the received second data. According to Structure 18, the same effect as Structure 1 is obtained.

[0034] The above objects, features, aspects and advantages of the present invention and other objects, features, aspects and advantages will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram showing a configuration example of a system according to this embodiment.

[0036] Figure 2 This is a diagram for explaining an example of imaging processing and display processing in the system according to this embodiment.

[0037] Figure 3 This is a diagram showing an example of processing results in the system according to this embodiment.

[0038] Figure 4 This is a diagram showing another example of the processing results in the system according to this embodiment.

[0039] Figure 5 This is a diagram showing an example of communication processing between a device and a camera in the system according to this embodiment.

[0040] Figure 6 This is a diagram showing another example of the communication process between the device and the camera in the system according to this embodiment.

[0041] Figure 7A and Figure 7B This is a diagram for explaining an example of embedding identification information in an image in the system according to this embodiment.

[0042] Figure 8A and Figure 8B This is a flowchart showing an example of a processing procedure in a device of the system according to this embodiment. DETAILED DESCRIPTION

[0043] The present embodiment will be described in detail with reference to the accompanying drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and their description will not be repeated.

[0044] [A. System configuration example]

[0045] First, a configuration example of the system 1 according to this embodiment will be described.

[0046] Reference Figure 1 , an example of the structure of a system 1 according to this embodiment is described. The system 1 includes a device 100 and a camera 200 connected to the device 100 by wire or wirelessly. The camera 200 can capture dynamic images. Alternatively, the camera 200 can be detachable relative to the device 100. The camera 200 can be, for example, a camera for video calls or a surveillance camera. Alternatively, a plurality of cameras 200 can be connected to the device 100. The camera 200 can be connected only to a specific device or to a variety of devices.

[0047] Device 100 can display images captured by camera 200. Device 100 includes, for example, processor 102, memory 104, storage device 110, input unit 106, display unit 108, and communication unit 120. Device 100 may be, for example, a personal computer, game device, television, smartphone, or the like.

[0048] The processor 102 is a processing entity (or processing unit) for executing processes in the device 100. The processor 102 includes, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 102 expands the program stored in the storage device 110 into the memory 104 and executes it.

[0049] The memory 104 is a volatile storage medium accessible to the processor 102 , and includes, for example, DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory).

[0050] The storage device 110 is a non-volatile storage medium accessible to the processor 102, including, for example, a flash memory and a hard disk. The storage device 110 may also include, for example, a removable storage medium such as a magnetic tape cartridge and an optical disk.

[0051] The storage device 110 stores, for example, a system program 112 and an application program 114 .

[0052] The system program 112 includes computer-readable commands for basic processing such as communication processing between the device 100 and the camera 200 , and processing of image data received from the camera 200 .

[0053] The application programs 114 include computer-readable commands for executing various application programs.

[0054] In this specification, the term "processor" includes at least processing circuits that execute processing according to computer-readable instructions, such as CPUs and GPUs, and hardwired circuits such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays).

[0055] In this specification, the term “memory” includes at least the memory 104 and the storage device 110 .

[0056] Device 100 may also employ a SoC (System on Chip) that integrates functions such as a processor and memory. In this case, the processor and memory reside in the same SoC (an example of an integrated circuit). Therefore, in this specification, the terms "processor" and "memory" encompass both independent and integrated configurations.

[0057] The input unit 106 receives user operations. Examples of the input unit 106 include a keyboard, mouse, game controller, buttons, a cross key, a microphone, and the like. The input unit 106 may also serve as an interface with a device that receives user operations. In other words, the device that receives user operations may also be external to the device 100.

[0058] Display unit 108 displays images or videos generated by processor 102. Examples of display unit 108 include an LCD (Liquid Crystal Display) and an organic EL display. Display unit 108 may also serve as an interface with a display device. In other words, the display device may be external to device 100.

[0059] The communication unit 120 transmits and receives data with the camera 200. The communication unit 120 can be connected to the camera 200 either wired or wirelessly. For a wired connection, for example, a USB (Universal Serial Bus) connection or a parallel connection can be used. For a wireless connection, for example, Bluetooth (registered trademark), ZigBee (registered trademark), and wireless LAN (IEEE 802.11 standard) can be used.

[0060] The device 100 may be a general-purpose personal computer. In this case, the device 100 may include a driver responsible for communicating with the camera 200 described later, in addition to or instead of the system program 112 .

[0061] The camera 200 includes, for example, an imaging element 210 , a communication unit 220 , and an image processing unit 230 .

[0062] The imaging element 210 outputs image data (hereinafter also referred to as "entire image data") based on incident light. The imaging element 210 includes, for example, a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. A lens may also be provided upstream of the imaging element 210. The range within which the imaging element 210 can generate image data is also referred to as the "imaging area."

[0063] The communication unit 220 transmits and receives data to and from the device 100. The communication unit 220 has the same configuration as that of the communication unit 120.

[0064] The image processing unit 230 processes the entire image data output from the imaging element 210 and transmits the processing result to the device 100 via the communication unit 220. The processing performed by the image processing unit 230 may be processing performed in accordance with a request and / or setting from the device 100. The image processing unit 230 may also include a buffer 232 for temporarily storing the entire image data output from the imaging element 210.

[0065] The image processing unit 230 may also include, for example, firmware 234. The firmware 234 includes computer-readable instructions for executing a process by the image processing unit 230. The image processing unit 230 may also include a processor, an ASIC, and an FPGA (not shown).

[0066] [B. Image Processing and Display Processing]

[0067] Next, the imaging process and the display process in the system 1 according to the present embodiment will be described.

[0068] Reference Figure 2 , an example of the imaging process and display process in the system 1 according to this embodiment is described. The image processing unit 230 of the camera 200 processes the image based on the imaging element 210 ( Figure 1 ) output, generates first image data 262 corresponding to the first area 252 in the imaging area 250 of the camera 200, and second image data 264 corresponding to the second area 254 in the imaging area 250 of the camera 200. For example, the image processing unit 230 alternately generates the first image data 262 and the second image data 264. When the image processing unit 230 generates images at 30 fps (frames per second), the first image data 262 may be generated at 15 fps, and the second image data 264 may be generated at 15 fps. The first area 252 and the second area 254 may also be set as ROIs (Regions of Interest). When the device 100 and the camera 200 are connected via USB and exchange data in accordance with the UVC (USB Video Class) standard, which is the communication standard of the camera, the ROI may also be set using a method in accordance with the UVC standard.

[0069] The communication unit 220 of the camera 200 sequentially transmits the image data generated by the image processing unit 230 to the device 100 .

[0070] Image data from camera 200, which is sequentially received by communication unit 120 of device 100, is stored in buffer 140. Buffer 140 may also include areas for separately storing first image data 262 and second image data 264. As first image data 262 and second image data 264 are sequentially updated in each area, buffer 140 stores the latest first image data 262 and the latest second image data 264.

[0071] The image synthesis unit 130 generates a synthesized image 170 including a first image 172 based on the first image data 262 stored in the buffer 140 and a second image 174 based on the second image data 264. The image synthesis unit 130 may also generate a synthesized image 170 based on the first image data 262 stored in the buffer 140. Figure 1 ) to generate the composite image 170. The generation cycle (or update cycle) of the composite image 170 may be shorter than the cycle of transmitting image data from the camera 200.

[0072] The image synthesis unit 130 outputs the generated synthesized image 170. The process of outputting the synthesized image 170 includes at least a part of the following processes: Figure 1 ) processing of displaying the composite image 170; processing of outputting a signal for displaying the composite image 170 to an external display device, etc.; processing of sending data for displaying the composite image 170 via a network, etc.; and processing of saving the data for displaying the composite image 170 to a storage medium.

[0073] Next, an example of the processing results in the system 1 according to this embodiment is shown. Figure 3 In the example shown, two people are present in the imaging area 250 of the camera 200. For example, a first area 252 is set to encompass the face of one person, and a second area 254 is set to encompass the face of the other person. In this case, the composite image 170 generated based on first image data 262 corresponding to the first area 252 and second image data 264 corresponding to the second area 254 includes images of the areas surrounding the faces of each person. In the composite image 170, the display position and size of the first image 172 and the second image 174 may or may not be pre-set. For example, the display position of at least one of the first and second images 172 and 174 may be moved depending on the execution status of the application 114.

[0074] The first region 252 and the second region 254 can each be arbitrarily set in the imaging region 250. Therefore, the entire imaging region 250 can be set as the first region 252 and / or the second region 254.

[0075] Another example of the processing result in the system 1 according to this embodiment is shown. Figure 4 In the example shown, a person is present in the imaging area 250 of the camera 200. For example, a first area 252 is set to the area around the person's face, and the entire imaging area 250 is set to a second area 254. In this case, the composite image 170 includes a first image 172 showing the area around the person's face and a second image 174 showing the entire field of view of the camera 200.

[0076] The first region 252 and the second region 254 can be set by any method.

[0077] The image synthesis unit 130 may also process at least one of the first image 172 and the second image 174. Figure 4 , the second image 174 includes a frame object 176 showing a region corresponding to the first image 172. That is, the image synthesis unit 130 generates the second image 174 by performing processing such that the frame object 176 is superimposed on the second image data 264 corresponding to the second region 254.

[0078] like Figure 4 As shown, by superimposing the frame object 176 showing the area corresponding to the first image 172, the user can clearly understand at a glance which area in the imaging area 250 of the camera 200 is zoomed. In addition, the user can clearly understand at a glance how the zoomed area is displayed.

[0079] [C. Communication Processing]

[0080] Next, a communication process between the device 100 and the camera 200 in the system 1 according to the present embodiment will be described.

[0081] In order for the camera 200 to transmit the first image data 262 and the second image data 264 , the device 100 issues a request and / or sets a setting to the camera 200 at an appropriate timing.

[0082] Reference Figure 5 , the device 100 will send the initial settings 180 required for the image data (in Figure 5 Initial settings 180 are sent to camera 200 (indicated as "Set Format") (sequence SQ2). Initial settings 180 include settings for the data format and / or image size of image data to be sent by camera 200. Camera 200 (or image processing unit 230) sets the default image size according to initial settings 180.

[0083] If the aspect ratio (e.g., 16:9, 3:4, etc.) is pre-set in the camera 200, the initial settings 180 may include only the number of pixels on the short side (e.g., 360 pixels, 180 pixels, etc.). If the aspect ratio is set to 16:9 and 360 pixels is set, the camera 200 sets the image size to 640 pixels × 360 pixels. The image size set in the initial settings 180 may also be smaller than the image size of the entire image data output by the imaging element 210 (e.g., 1920 pixels × 1080 pixels). In other words, the image size set in the initial settings 180 may be smaller than the standard image size.

[0084] The period (fps) at which the camera 200 transmits image data may be set in advance or may be included in the initial setting 180 .

[0085] Next, the device 100 sets the first area setting 182 (in Figure 5 The first region 252 is set as the ROI according to the first region setting 182 (indicated as “Set ROI1” in FIG. 1 ).

[0086] The first area setting 182 includes a request for setting the image data included in the data transmitted by the camera 200 to the first image data 262 corresponding to the first area 252 in the imaging area 250 .

[0087] Next, the camera 200 begins generating image data for the set ROI in synchronization with the transmission cycle (sequence SQ6). When the first region 252 is set as the ROI, the camera 200 (or the image processing unit 230) generates first image data 262 corresponding to the first region 252. The camera 200 (or the image processing unit 230) transmits data including the generated first image data 262 to the device 100 (sequence SQ8).

[0088] The device 100 sets the second area 254 as the area where image data should be generated (in the second area setting 184). Figure 5 The image processing unit 230 sets the second region 254 as the ROI according to the second region setting 184 (indicated as “Set ROI2” in FIG. 1 ).

[0089] The second area setting 184 includes a request to set the image data included in the data transmitted by the camera 200 to the second image data 264 corresponding to the second area 254 in the imaging area 250 .

[0090] Next, the camera 200 begins generating image data for the set ROI in synchronization with the transmission cycle (sequence SQ12). When the second region 254 is set as the ROI, the camera 200 (or the image processing unit 230) generates second image data 264 corresponding to the second region 254. The camera 200 (or the image processing unit 230) transmits data including the generated second image data 264 to the device 100 (sequence SQ14).

[0091] Thereafter, the same processing as in sequences SQ4 to SQ14 is repeatedly executed.

[0092] like Figure 5 As shown, by the device 100 sequentially providing the first area setting 182 and / or the second area setting 184 (hereinafter sometimes collectively referred to as "area setting") for generating image data to the camera 200, the device 100 is able to obtain image data corresponding to different areas in the camera area 250.

[0093] In this manner, the device 100 alternately switches and transmits the first area setting 182 and the second area setting 184 to the camera 200. The first area setting 182 is used to cause the camera 200 to transmit image data corresponding to the first area 252 in the imaging area 250 of the camera 200, and the second area setting 184 is used to cause the camera 200 to transmit image data corresponding to the second area 254 in the imaging area 250. As will be described later, the first area setting 182 and the second area setting 184 do not necessarily need to be switched in a strictly alternating manner.

[0094] The camera 200 follows the first area setting 182 and uses the overall image data corresponding to the imaging area 250 (eg, Figure 1 The camera 200 generates first image data 262 corresponding to the first area 252 based on the image data output by the imaging element 210 shown in FIG. 1 and transmits data including the first image data 262 to the device 100. The camera 200 also generates second image data 264 corresponding to the second area 254 based on the overall image data in accordance with the second area setting 184 and transmits data including the second image data 264 to the device 100.

[0095] Furthermore, in sequences SQ6 and SQ12, camera 200 may also start generating image data for the set ROI based on an image transmission request from device 100. In this case, device 100 sequentially transmits image transmission requests to camera 200 in synchronization with the transmission cycle. In other words, camera 200 may transmit data including the set image data in response to the image transmission request, which is a request for camera 200 to transmit data including the image data.

[0096] Therefore, the “first request” can include the first area setting 182 and an image transmission request for requesting image data corresponding to the first area setting 182. Similarly, the “second request” can include the second area setting 184 and an image transmission request for requesting image data corresponding to the second area setting 184.

[0097] like Figure 5 As shown, the device 100 transmits the initial setting 180 for setting the image size of the image data transmitted by the camera 200 to the camera 200 before transmitting the first area setting 182 and the second area setting 184 .

[0098] Next, another example of the communication process between the device 100 and the camera 200 in the system 1 according to the present embodiment will be described. Figure 6 An example is shown in which the device 100 transmits the first area setting 182 to the camera 200 after a time period exceeding one transmission cycle has elapsed since the device 100 transmitted the second area setting 184 to the camera 200 .

[0099] exist Figure 6 In the example shown, after camera 200 sets second region 254 as the ROI according to second region setting 184, it maintains the previously set ROI until a new region setting is received. Therefore, camera 200 transmits second image data 264 twice in succession. That is, while camera 200 alternately transmits first image data 262 and second image data 264, it can also transmit either image data continuously.

[0100] As such, as long as the device 100 can acquire the first image data 262 and the second image data 264 within a period such that the user can recognize that the first image 172 and the second image 174 were captured at the same time to a certain extent, the transmission of the first image data 262 and the transmission of the second image data 264 do not need to be strictly alternating. In other words, the transmission order and transmission mode when the first image data 262 and the second image data 264 are alternately transmitted can be arbitrarily set and changed according to the circumstances of the device 100. For example, the transmission frequency of one of the first image data 262 and the second image data 264 can be increased.

[0101] [D. Regional Settings]

[0102] Next, the area setting transmitted from the device 100 to the camera 200 will be described.

[0103] The first area setting 182 and the second area setting 184 may also include identification information for specifying each area setting. For example, the first area setting 182 may include "0" as identification information, and the second area setting 184 may include "1" as identification information. Alternatively, if there are two area settings, identification information may be assigned to only one of the area settings. For example, the first area setting 182 may include identification information, while the second area setting 184 may not include identification information.

[0104] Thus, the first area setting 182 may include the first identification information, and the second area setting 184 may include third identification information different from the first identification information. The identification information included in the first area setting 182 and the identification information included in the second area setting 184 may be either of two values.

[0105] Furthermore, the settings of first area setting 182 and second area setting 184 do not need to be associated with the values ​​of the identification information included in first area setting 182 and second area setting 184. The identification information is used to identify which of the multiple area settings is being used, and does not necessarily specify the content of the area setting. In other words, the identification information included in first area setting 182 and the identification information included in second area setting 184 can be set regardless of which area in imaging area 250 the camera 200 is to transmit image data corresponding to.

[0106] Furthermore, as will be described later, the camera 200 can determine identification information to be added to generated image data based on the identification information included in the area setting (and / or the absence of identification information).

[0107] By adopting such identification information, for example, even when the setting contents of the first area setting 182 and the setting contents of the second area setting 184 are the same, the device 100 can determine which area setting the image data corresponds to. As an example, consider the following case: when the first area 252 and the second area 254 change relative to each other, the first area 252 and the second area 254 become identical. In this case, there is a concern that the device 100 will not be able to determine which of the first image 172 and the second image 174 in the composite image 170 should be updated based on the image data sent from the camera 200. However, if identification information is adopted as described above, even when the first area 252 and the second area 254 become identical, it can be clearly known which of the first image 172 and the second image 174 should be updated based on the image data sent from the camera 200. In addition, in other embodiments, identification information may not be used. As an example, when first area 252 and second area 254 are the same, first image 172 and second image 174 , that is, first image data 262 and latest second image data 264 stored in buffer 140 , may be updated based on received image data.

[0108] The first region setting 182 and the second region setting 184 may also include, as parameters for setting the ROI, the coordinates (e.g., X and Y coordinates) of the ROI's reference position (e.g., the top left vertex) and the number of pixels of one side (e.g., the long side) of the ROI. If the aspect ratio is already set, the image size of the ROI is calculated by setting the number of pixels of one side of the ROI.

[0109] For example, if (100, 100, 480) is set as the parameters (e.g., X coordinate, Y coordinate, and number of pixels on the long side) for the first region setting 182 or the second region setting 184, and the aspect ratio is 16:9, the image size is calculated as 480 pixels x 270 pixels. Consequently, a quadrilateral region with vertices at coordinates (100, 100), (100, 370), (580, 100), and (580, 370) is set as the ROI.

[0110] The image processing unit 230 of the camera 200 extracts image data corresponding to the region set as the ROI from the entire image data output from the imaging element 210. Alternatively, rather than temporarily storing the entire image data output from the imaging element 210 and then extracting the image data, the imaging element 210 may directly output the image data corresponding to the region set as the ROI.

[0111] In the above example, the extracted image data is 480 pixels × 270 pixels. Therefore, the image processing unit 230 of the camera 200 scales (e.g., reduces or enlarges) the extracted image to match the image size set in the initialization 180 (e.g., 640 pixels × 360 pixels). The image processing unit 230 of the camera 200 may also scale up (or increase the resolution) the extracted image. If the image size of the ROI matches the image size set in the initialization 180, the scaling process may be omitted.

[0112] The image size of the image data generated by the camera 200 is preset in the initial setting 180. In addition, the image processing unit 230 of the camera 200 scales the image. Therefore, the first area setting 182 or the second area setting 184 does not need to include a request to change the image size.

[0113] As described above, in system 1 according to this embodiment, since camera 200 generates image data required for display in device 100, unnecessary image data does not need to be transmitted, and the communication bandwidth between device 100 and camera 200 can be effectively utilized. Furthermore, image data of a predetermined image size is transmitted regardless of the set ROI position and size, thereby suppressing fluctuations in the processing load for display in device 100.

[0114] [E. Recognition of Image Data]

[0115] Next, an example of a method for the device 100 to identify to which ROI the image data received from the camera 200 corresponds will be described.

[0116] (e1: Storage of identification information in the header)

[0117] Identification information for specifying the corresponding ROI may be included in the data transmitted from the camera 200 to the device 100 .

[0118] Specifically, the first data including the first image data 262 transmitted from the camera 200 may also include identification information corresponding to the identification information of the first area setting 182. Furthermore, the second data including the second image data 264 transmitted from the camera 200 may also include identification information corresponding to the identification information of the second area setting 184. If the second area setting 184 does not include identification information, the second data including the second image data 264 transmitted from the camera 200 may also not include identification information. The first and second identification information may be the same or different, and the third and fourth identification information may be the same or different.

[0119] The identification information included in the data may be in any format. For example, the identification information used to determine the ROI may be stored in a header defined in the UVC standard.

[0120] The camera 200 may directly include identification information (for example, “0” or “1”) of the area setting corresponding to the generated image data in the header.

[0121] If there is no identification information for the region setting corresponding to the generated image data, the identification information may not be included in the header. If there is identification information for the region setting corresponding to the generated image data, information indicating the existence of the setting information may be included in the header. For example, if a partial region and the entire region of the imaging area 250 are each set as an ROI, identification information indicating that the region setting has been performed (or identification information identical to the identification information for the region setting) may be added to the image data representing the partial region, while no identification information may be added to the image data representing the entire region.

[0122] Alternatively, the camera 200 may include in the header a value (eg, a character string such as “ROI1” or “ROI2”) calculated according to a predetermined rule based on identification information set for the region corresponding to the generated image data.

[0123] The identification information stored in the header can also be described in key-value form.

[0124] As described above, the device 100 can determine which region setting the received image data corresponds to based on the identification information included in the data received from the camera 200. This configuration stabilizes the process of determining the ROI corresponding to the image data in the device 100.

[0125] (e2: Embedding of identification information in the image)

[0126] Identification information for identifying which area setting the image is based on may be embedded in the image. In other words, identification information for identifying the corresponding area setting may be stored in the image data.

[0127] After generating image data, the camera 200 rewrites specific pixel values ​​of the generated image data based on the region setting corresponding to the generated image data.

[0128] Reference Figure 7A and Figure 7B , an example of embedding identification information in an image in the system 1 according to this embodiment is described. As an example, Figure 7A and Figure 7B A diagram showing brightness (Y) in YCbCr image data.

[0129] The identification information may also be stored in the data corresponding to the image of the edge in the ROI in the image data. For example, the specific coordinates of the image data (in Figure 7A and Figure 7B In the example shown, the pixel value (or brightness value) of the upper left vertex is forcibly set to "0" ( Figure 7A ) or "1" ( Figure 7B ). It can also be interpreted as: “0” represents an image generated according to the first area setting 182, and “1” represents an image generated according to the second area setting 184.

[0130] When receiving image data in which identification information is embedded from the camera 200 , the device 100 determines, based on the pixel values ​​at specific coordinates, which region setting the received image data is based on.

[0131] Although the pixel values ​​at specific coordinates in the received image data are different from the original values, if the image size of the image data is large enough, the visual unnaturalness can be ignored, so the image data can be directly used to generate the synthesized image 170. Alternatively, the device 100 can estimate the pixel value at the specific coordinate based on the pixel values ​​located around the specific coordinate and correct it.

[0132] The pixel position where the identification information is embedded may be any position. Even when image data is generated in any format other than YCbCr, such as RGB or CMYK, the identification information can be embedded in the image in the same manner.

[0133] By adopting a method of embedding identification information in an image, the device 100 can determine which region setting corresponds to received image data even without storing the identification information in a header or the like.

[0134] (e3: Timing-based recognition)

[0135] As described above, the device 100 provides the camera 200 with the region setting, thereby setting the ROI in the camera 200, and the camera 200 transmits image data corresponding to the set ROI. Therefore, when the device 100 receives image data from the camera 200, if a predetermined delay time has elapsed since the timing of receiving the image data, the device 100 may determine the latest region setting and determine that the image data corresponds to the determined region setting.

[0136] The predetermined delay time may be determined in consideration of the time required to set the ROI in the camera 200 , the time required to generate image data corresponding to the ROI in the camera 200 , and the time required to transmit the image data from the camera 200 to the device 100 .

[0137] [F. Processing in Device 100]

[0138] Next, an example of processing in the device 100 will be described.

[0139] Figure 8A and Figure 8B The steps shown may also be implemented by the processor 102 of the device 100 executing the system program 112 and / or the application program 114 . Figure 8A 2 shows a process related to a request from the device 100 to the camera 200 . Figure 8B 2 shows a process related to reception of data from the camera 200 in the device 100 .

[0140] Reference Figure 8A When an image display request is generated ("Yes" in step S100), the device 100 sends the initial setting 180 to the camera 200 (step S102). The image display request may be issued by a user operation or by the application 114 (or Figure 2 The application execution unit 150 shown in FIG.

[0141] Next, the device 100 transmits the first area setting 182 to the camera 200 in synchronization with the transmission cycle (step S104). Next, the device 100 transmits the second area setting 184 to the camera 200 in synchronization with the transmission cycle (step S106).

[0142] The device 100 repeatedly performs the processing of steps S104 and S106 until being instructed to stop image display.

[0143] When instructed to stop the image display ("Yes" in step S108), the device 100 ends the process. The device 100 may end the process after sending a request to the camera 200 to reset the settings.

[0144] Alternatively, after steps S104 and S106 , the device 100 may send an image transmission request to the camera 200 .

[0145] Reference Figure 8B When an image display request is generated ("Yes" in step S100), the device 100 determines whether data has been received from the camera 200 (step S120). When data has been received from the camera 200 ("Yes" in step S120), the device 100 determines whether the received data includes the first image data 262 or the second image data 264 (step S122), and stores the image data included in the received data in the corresponding area of ​​the buffer 140 (step S124).

[0146] In step S122 , it is determined which region setting the image data corresponds to by any of the above methods.

[0147] Next, device 100 generates composite image 170 based on first image data 262 and second image data 264 stored in buffer 140 (step S126). Device 100 then outputs generated composite image 170 to display unit 108 (step S128). If only one of first image data 262 and second image data 264 is stored in buffer 140, steps S126 and S128 may be skipped.

[0148] The device 100 repeats the processing of steps S120 to S128 until it is instructed to stop the image display.

[0149] When instructed to stop the image display (YES in step S130 ), the device 100 ends the process.

[0150] Figure 8A and Figure 8B The illustrated processing may be executed in parallel or serially. Figure 8A and Figure 8B The execution order of the processes shown can also be changed arbitrarily. Figure 8A or Figure 8B Multiple steps shown may also be performed in parallel. Figure 8A or Figure 8B Multiple steps shown may also be performed as one step.

[0151] The device 100 arranges the first image 172 based on the first image data 262 and the second image 174 based on the second image data 264 at predetermined positions of the composite image 170 (see Figure 3 and Figure 4 ). Based on the fact that the first data received from the camera 200 (e.g., data including the first image data 262) includes identification information indicating the first area setting 182, the device 100 arranges the first image 172 based on the first image data 262 at a predetermined position (a first position: for example, the right side) in the composite image 170. Similarly, based on the fact that the second data received from the camera 200 (e.g., data including the second image data 264) includes identification information indicating the second area setting 184, the device 100 arranges the second image 174 based on the second image data 264 at another predetermined position (a second position: for example, the left side) in the composite image 170.

[0152] Alternatively, in the case where the second area setting 184 does not include identification information, the device 100 may also arrange the second image 174 based on the second image data 264 at another specified position (second position: for example, the left side) in the composite image 170 based on the fact that the second data received from the camera 200 (for example, data including the second image data 264) does not include identification information representing the second area setting 184.

[0153] [G. Modifications]

[0154] The above embodiment shows an example of preparing two areas for generating image data. However, three or more areas may be set in an alternating manner. In this case, the composite image 170 includes three or more images. In this case, it can be said that the request for the camera to transmit image data corresponding to two of the three or more areas is alternately switched.

[0155] The area for generating image data can also be determined by any method. For example, it can be pre-set, user-set, or the ROI can be set around a person's face based on the results of facial recognition. When determined by facial recognition, by sequentially performing facial recognition, the ROI can track the area around the person's face even if the area containing the person in imaging area 250 changes over time. Such facial recognition processing can be performed by either device 100 or camera 200. As an example, device 100 can also determine the location of a person's face by performing image processing on received image data and generate a region setting corresponding to the determined location. On the other hand, when facial recognition processing is performed by camera 200, device 100 can also send a request for facial authentication (e.g., a request to recognize the face of a specific person in imaging area 250) to camera 200. When information about the range of the ROI set by device 100 is required, camera 200 can also send the range of the ROI corresponding to the image data (e.g., the coordinates of the vertices) to device 100 in addition to the generated image data.

[0156] The above Figure 3 and Figure 4 , an example is shown in which the device 100 displays images based on respective image data corresponding to a plurality of areas sent from the camera 200, but the plurality of images based on the plurality of image data may not always be displayed. In the case where the device 100 does not display a plurality of images based on respective image data corresponding to a plurality of areas (i.e., displays only images based on image data corresponding to a specific area), the shooting frequency of the image data used for display may be made higher than the shooting frequency of the image data in the case of shooting a plurality of image data corresponding to a plurality of areas. For example, the device 100 may sometimes display only an image showing the entire imaging area 250 (e.g., displays only an image showing the entire imaging area 250). Figure 4 The second image 174 shown in FIG. 174 is also shown. At this time, ROI may not be set. Figure 4 The composite image 170 shown includes the first image 172 and the second image 174. When the area to be zoomed is further determined, only the image 172 and the second image 174 may be displayed. Figure 4 A first image 172 is shown.

[0157] The data transmitted from camera 200 can include the generated image data directly (i.e., in an uncompressed state) or compressed image data. When the image data is transmitted in an uncompressed state, compression processing in camera 200 and decompression processing in device 100 are unnecessary, thereby reducing the processing load. When the image data is compressed and transmitted, a larger amount of other data can be transmitted.

[0158] Camera 200 may also be a general-purpose camera. However, a general-purpose camera may not be able to transmit data containing identification information. In such cases, the general-purpose camera may transmit data containing an ROI to device 100. Device 100 may also determine the ROI corresponding to the received image data based on the ROI value contained in the received data, and generate composite image 170 based on the determined ROI.

[0159] The ROI may be set by a method according to the UVC standard as described above, or may be set by a method according to any communication standard.

[0160] As an example of processing at least one of first image 172 and second image 174, device 100 may also generate an avatar based on at least one of first image data 262 and second image data 264. For example, an avatar may be generated that moves in accordance with the movements of the subject (user) in first image data 262 or second image data 264, or an avatar may be generated whose expression changes in accordance with the expression of the subject (user). By setting the area required for control of avatar generation, device 100 can achieve efficient data transmission.

[0161] Device 100 outputs the generated composite image 170 to display unit 108, but may also output it to other output destinations. For example, device 100 may output composite image 170 to a transmission unit to transmit it to another device 100, and then display composite image 170 or an image processed from composite image 170 on the other device 100. Furthermore, device 100 may transmit at least one of first image data 262 and second image data 264, or image data based on these image data, to the other device 100. The other device 100 may also generate an image using these or other image data and display the image.

[0162] While the embodiments of the present disclosure have been described, they should be considered in all respects to be illustrative and non-restrictive. The scope of the present disclosure is indicated by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

Claims

1. A system comprising: equipment; and A camera is connected to the device in a wired or wireless manner. in, The device alternately switches and sends a first request and a second request to the camera, wherein the first request is used to cause the camera to send image data corresponding to a first area in a camera area, and the second request is used to cause the camera to send image data corresponding to a second area in the camera area. The camera generates first image data corresponding to the first area based on the overall image data corresponding to the camera area according to the first request, and sends first data including the first image data to the device. The camera generates second image data corresponding to the second area based on the overall image data according to the second request, and sends second data including the second image data to the device. The device outputs a composite image including a first image based on the first image data included in the received first data and a second image based on the second image data included in the received second data.

2. The system according to claim 1, wherein: The first request includes first identification information, and the first data includes second identification information corresponding to the first identification information. The device arranges the first image at a first position in the composite image based on the received first data including the second identification information. The device arranges the second image at a second position in the composite image based on the fact that the received second data does not include the second identification information.

3. The system according to claim 1, wherein: The first request includes first identification information, and the first data includes second identification information corresponding to the first identification information. The second request includes third identification information different from the first identification information, and the second data includes fourth identification information corresponding to the third identification information and different from the second identification information. The device arranges the second image at a second position in the composite image based on the fact that the received second data includes the fourth identification information.

4. The system according to claim 3, wherein: The first identification information and the third identification information are set regardless of which area of ​​the imaging area the camera is caused to transmit image data.

5. The system according to claim 3, wherein: The first identification information and the third identification information are either one of two values.

6. The system according to any one of claims 2 to 5, wherein: The first identification information is stored in a header defined in the USB video class standard, ie, the UVC standard.

7. The system according to any one of claims 2 to 5, wherein: The first identification information is stored in the first image data.

8. The system according to claim 7, wherein: The first identification information is stored in data corresponding to an image of an edge in the first area in the first image data.

9. The system according to any one of claims 1 to 5, wherein: The first request includes a request for setting image data included in data transmitted by the camera to image data corresponding to the first area in the imaging area. The second request includes a request for setting image data included in data transmitted by the camera to image data corresponding to the second area in the imaging area. The camera transmits data including the set image data in response to a third request that is a request for the camera to transmit data including the image data.

10. The system according to any one of claims 1 to 5, wherein: The device sends a fourth request for setting an image size of image data to be sent by the camera before sending the first request and the second request. The first request and the second request do not include a request to change the image size.

11. The system according to claim 10, wherein: The image size for which the setting is performed is an image size smaller than the image size in the standard.

12. A camera connected to a device in a wired or wireless manner, wherein: The device alternately switches and sends a first request and a second request to the camera, wherein the first request is used to cause the camera to send image data corresponding to a first area in a camera area, and the second request is used to cause the camera to send image data corresponding to a second area in the camera area. The camera generates first image data corresponding to the first area based on the overall image data corresponding to the camera area according to the first request, and sends first data including the first image data to the device. The camera generates second image data corresponding to the second area based on the overall image data according to the second request, and sends second data including the second image data to the device. The first request includes first identification information, and the first data includes second identification information corresponding to the first identification information.

13. The camera according to claim 12, wherein The first identification information is stored in a header defined in the USB video class standard, ie, the UVC standard.

14. The camera according to claim 12, wherein The first identification information is stored in the first image data.

15. A device connected to a camera in a wired or wireless manner, wherein: The device alternately switches and sends a first request and a second request to the camera, wherein the first request is used to cause the camera to send image data corresponding to a first area in a camera area, and the second request is used to cause the camera to send image data corresponding to a second area in the camera area. The camera generates first image data corresponding to the first area based on the overall image data corresponding to the camera area according to the first request, and sends first data including the first image data to the device. The camera generates second image data corresponding to the second area based on the overall image data according to the second request, and sends second data including the second image data to the device. The device outputs a composite image including a first image based on the first image data included in the received first data and a second image based on the second image data included in the received second data.

16. The apparatus according to claim 15, wherein The first request includes first identification information, and the first data includes second identification information corresponding to the first identification information. The device arranges the first image at a first position in the composite image based on the received first data including the second identification information. The device arranges the second image at a second position in the composite image based on the fact that the received second data does not include the second identification information.

17. The apparatus according to claim 15, wherein The first request includes first identification information, and the first data includes second identification information corresponding to the first identification information. The second request includes third identification information different from the first identification information, and the second data includes fourth identification information corresponding to the third identification information and different from the second identification information. The device arranges the second image at a second position in the composite image based on the fact that the received second data includes the fourth identification information.

18. A method, performed in a system comprising a device and a camera connected to the device in a wired or wireless manner, comprising the following steps: The device alternately switches and sends a first request and a second request to the camera, wherein the first request is used to cause the camera to send image data corresponding to a first area in a camera area, and the second request is used to cause the camera to send image data corresponding to a second area in the camera area; The camera generates first image data corresponding to the first area based on the overall image data corresponding to the camera area according to the first request, and sends first data including the first image data to the device; the camera generating second image data corresponding to the second area based on the overall image data in accordance with the second request, and sending second data including the second image data to the device; and The device outputs a composite image including a first image based on the first image data included in the received first data and a second image based on the second image data included in the received second data.

Images