Imaging control device and imaging control method

Abstract

To provide a technique for determining an angle of view including interrelated descriptions on an information sharing board.SOLUTION: The continuity between a description added to an imaging subject from a frame in which the imaging subject was imaged and a description that has been added to the imaging subject before the description is determined. On the basis of the determination result, descriptions of the imaging subject are divided into groups. On the basis of the time on each group, the angle of view of imaging apparatus is controlled to image the description of the imaging subject of a group to be distributed in the frame.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to imaging control technology. [Background technology]

[0002] In lectures and other information-sharing settings, information boards such as blackboards, whiteboards, and screens are commonly used to help participants understand the content. Therefore, when broadcasting a lecture video to participants outside the lecture hall, it is desirable to include the information displayed on the information board in the broadcast video. In this case, if the viewing angle is too wide, the text in the broadcast video will be too small and difficult to read, while if the viewing angle is too narrow, necessary information may not be included in the broadcast video, making it difficult to understand the lecture. Therefore, it is important to capture the information board with an appropriate viewing angle.

[0003] Meanwhile, with the aim of reducing manpower, several technologies have been proposed for determining an appropriate angle of view for streaming video without requiring cameraman operation. Patent Document 1 detects changes in the feature quantities of a preset imaging area and selectively determines the cropping range. Patent Document 2 determines an angle of view for panning while maintaining a zoom ratio that allows text to be read in accordance with the description. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 087577 [Patent Document 2] Patent No. 5398359 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is no established method for recognizing the relevance of information on an information sharing board, like a cameraman, and appropriately controlling the angle of view each time. Even when using the method disclosed in Patent Document 1, if there is a closely related blackboard note located far from the instructor, the video could not be simultaneously distributed to remote participants. Similarly, even when using the method disclosed in Patent Document 2, only the area where the instructor last wrote was captured, and any related blackboard notes located far away were not included in the image capture. Therefore, if there is no positional continuity between the latest and previous notes, remote participants could not simultaneously view the closely related blackboard notes. This disclosure provides a technology for determining the angle of view that includes closely related notes on an information sharing board. [Means for solving the problem]

[0006] One aspect of the present disclosure is characterized by comprising a grouping means that determines the continuity of a description added to an imaging target and a description that was added to the imaging target earlier from a frame capturing an image of the imaging target, and groups the descriptions in the imaging target based on the result of the determination, and a control means that controls the angle of view of an imaging device so as to capture the description of the imaging target of the group to be distributed in the frame, based on the time for each group obtained by the grouping means. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide a technique for determining an angle of view that includes descriptions on an information sharing board that are closely related to each other. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of a system configuration. [Figure 2] FIG. 2 is a block diagram showing an example of the functional configuration of an imaging control device 101. [Figure 3] 10 is a flowchart of a grouping process for descriptions on an information sharing board. [Figure 4] FIG. 10 is a diagram schematically illustrating a method for detecting the size of a character. [Figure 5] 10 is a flowchart of a grouping process related to projection onto an information sharing board. [Figure 6] 10 is a flowchart of a time update process. [Figure 7] 10 is a flowchart of a target position recording process. [Figure 8] 10 is a flowchart of a target position update process. [Figure 9] FIG. 4 is a diagram showing an example of the configuration of description information. [Figure 10] FIG. 10 is a diagram showing the contents represented by the description information in FIG. 9. [Figure 11] 10 is a flowchart of a process for determining an angle of view. [Figure 12] 10 is a flowchart of a view angle composition process. [Figure 13] 5A and 5B are diagrams illustrating a view angle composition process. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the present disclosure according to the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the present disclosure, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [First embodiment] First, an example of the configuration of a system according to this embodiment will be described with reference to the block diagram of FIG. 1. The system according to this embodiment is a system that captures video of a lecture being given using a flat member such as a blackboard, whiteboard, or screen (hereinafter referred to as an information sharing board), and distributes the captured video. As shown in FIG. 1, the system according to this embodiment includes an imaging device 102, an imaging control device 101, and an audio-visual terminal 104. The imaging control device 101 and the audio-visual terminal 104 are configured to be able to communicate data with each other via a wired and / or wireless network 103. Note that the network configuration between the imaging control device 101 and the audio-visual terminal 104 is not limited to a specific network configuration.

[0011] First, the imaging device 102 will be described. The imaging device 102 is installed and set in a lecture room where information sharing boards are installed so that it can capture images of all information sharing boards in the lecture room. The imaging device 102 has one high-resolution camera, which captures video of "a lecture being given using the information sharing board," and outputs each frame of the captured video sequentially to the imaging control device 101. The imaging device 102 and the imaging control device 101 may be connected by wire or wirelessly.

[0012] Next, the imaging control device 101 will be described. The imaging control device 101 determines the range (angle of view) of an area (low-image-quality area) to be cut out from a frame (high-resolution frame) received from the imaging device 102, cuts out the area of ​​the determined angle of view, and generates a frame for distribution that includes the cut-out area. In the following description, the zoom ratio of the angle of view refers to the magnification when enlarging or reducing the cut-out area to fit the frame for distribution. The imaging control device 101 then distributes the generated frame for distribution to the viewing terminal 104 via the network 103. Note that, although audio will not be mentioned below, it is assumed that the frame distributed by the imaging control device 101 to the viewing terminal 104 also includes audio. However, it is not essential that the frame include audio, and it is not necessary for it to include audio.

[0013] The CPU 201 executes various processes using computer programs and data stored in the storage unit 202. As a result, the CPU 201 controls the overall operation of the imaging control device 101, and also executes or controls various processes that will be described as being performed by the imaging control device 101.

[0014] The storage unit 202 is a non-volatile memory, and stores computer programs and data for causing the CPU 201 to execute or control various processes that will be described as being performed by the imaging control device 101.

[0015] The interface 203 includes an interface for connecting the imaging control device 101 to the network 103 and an interface for connecting the imaging control device 101 to the imaging device 102 .

[0016] Next, we will explain the viewing terminal 104. The viewing terminal 104 displays / plays the frames for distribution transmitted from the imaging control device 101 via the network 103. This allows students who are remotely attending a lecture being captured by the imaging device 102 to use the viewing terminal 104 to view the lecture (video stream).

[0017] An example of the functional configuration of the imaging control device 101 is shown in the block diagram of Fig. 2. In this embodiment, the functional units shown in Fig. 2 are implemented by computer programs. In the following, the functional units shown in Fig. 2 may be described as the subject of processing, but in reality, the functions of the corresponding functional units are realized by the CPU 201 executing the computer programs corresponding to the functional units. Note that one or more of the functional units shown in Fig. 2 may be implemented by hardware.

[0018] Next, a description will be given of the process of determining the angle of view performed by the imaging control device 101. The process of determining the angle of view performed by the imaging control device 101 is roughly divided into a first stage, an interruption, and a second stage. First, the first and second stage processes will be described.

[0019] In the first stage of processing, frames received from the image capture device 102 are analyzed to generate information to be used when determining the angle of view in the second stage of processing. In the middle stage of processing, the information generated by the first stage of processing is stored in the storage unit 202. Furthermore, as the first and second stages of processing are executed, the stored information is read out as necessary and provided to each process.

[0020] The "generation of information" in the preceding process may include a grouping process for descriptions on the information sharing board, a grouping process for projection onto the information sharing board, a time update process based on instructions for descriptions and projections on the information sharing board, a target position recording process and a target position updating process for the information sharing board. In other words, the "generation of information" may include all or some of the "grouping process for descriptions on the information sharing board, a grouping process for projection onto the information sharing board, a time update process based on instructions for descriptions and projections on the information sharing board, a target position recording process and a target position updating process for the information sharing board." Next, each of the above processes that may be included in the "generation of information" will be described.

[0021] <Grouping processing of descriptions on information sharing boards> The grouping process for the descriptions on the information sharing board is a process that is performed in response to a description added to the information sharing board by the instructor. The grouping process for the descriptions on the information sharing board will be described with reference to the flowchart of FIG.

[0022] In step S301, the description detection unit 10 detects a description (such as a character or a line drawing) newly added to the information sharing board from a frame captured by the imaging device 102. The description detection can be performed by a known technique, for example, a technique based on background difference using an image or a technique based on character recognition of an image.

[0023] In step S302, the description analysis unit 11 determines whether the description detected in step S301 has continuity with a description that was added to the information sharing board before (in the past) the description. There are various methods for determining whether there is continuity between descriptions.

[0024] For example, if the difference (time interval) in the detection timing between a newly detected description (description A) and a description (description B) that was detected more recently than description A is within a certain time, it is determined that there is continuity (temporal continuity) between description A and description B.

[0025] For example, if the distance (spatial distance) between a newly detected description (description A) and a description (description B) that was detected more recently than description A is within a certain distance, it is determined that there is continuity (spatial continuity) between description A and description B.

[0026] In other cases, i.e., when a certain amount of time has passed since the detection of description A after the detection of description B, or when no past description is within a certain distance from description A, it may be determined that there is no continuity.

[0027] Furthermore, for example, the continuity between descriptions may be determined by taking into consideration both the temporal relationship and the spatial relationship between the descriptions. For example, both the time interval and the spatial distance may be scored, and continuity may be determined if the weighted average exceeds a threshold.

[0028] Furthermore, the index for determining whether or not there is continuity between descriptions may be one or more. As such, there are various methods for determining whether or not there is continuity between descriptions, and the method is not limited to a specific method.

[0029] If it is determined that there is continuity between the description detected in step S301 and the description already existing on the information sharing board, the process proceeds to step S305. On the other hand, if it is determined that there is no continuity between the description detected in step S301 and the description already existing on the information sharing board, the process proceeds to step S303.

[0030] In step S303, the description analysis unit 11 creates a new group for the description detected in step S301. When the process proceeds from step S303 to step S304, in step S304 the description generation unit 18 generates description information for the group created in step S303 based on the description detected in step S301.

[0031] On the other hand, in step S305, the description analysis unit 11 identifies a group of descriptions that are determined to have continuity with the description detected in step S301. If the process proceeds from step S305 to step S304, in step S304, the description generation unit 18 updates the description information of the group identified in step S305 based on the description detected in step S301 so that the group identified in step S305 includes the description detected in step S301.

[0032] The description information generated by the description generation unit 18 for each group includes the position of the group, the allowable zoom range for the group, and the time for the group. The position of the group is position information that defines a rectangle (group area) that surrounds the descriptions included in the group. The position information that defines the rectangle may be the positions of the four corners of the rectangle, the positions of the upper left corner and the lower right corner of the rectangle, or the position of the upper left corner of the rectangle and the height and width of the rectangle.

[0033] The allowable zoom range for a group is a constraint on the zoom range when including the group in the angle of view, and is used when determining the angle of view later. The time for a group is logical time, and is information that can identify the order of generation / update between groups. The time for a group may be the actual time (real time) when the group was generated / updated, or it may be something that determines the order relationship between groups, such as a counter value that is counted up when a group is generated / updated.

[0034] The description analysis unit 11 calculates the allowable zoom range for a group based on the size of the description included in the group. The method for calculating the size of the description differs depending on whether the description includes text. A known character recognition technique can be used to determine whether the description includes text. If the description includes text, the allowable zoom range is calculated based on the size of the text and information about the viewing terminal 104 (such as the screen resolution of the viewing terminal 104). The lower limit of the zoom range is the zoom factor at the maximum zoom out range where text is readable on the screen of the viewing terminal 104. The upper limit of the zoom range is the zoom factor at the maximum zoom in range where the entire group fits within the screen. On the other hand, if the description does not include text, a predetermined value is used as the allowable zoom range. The predetermined value may be expressed as a range or a single value. A single value indicates that the upper limit and lower limit of the zoom range are equal. Alternatively, the allowable zoom range may be calculated for each group using a predetermined function that inputs the properties of each group, such as the size of a rectangle enclosing the group.

[0035] To detect character size, a lightweight method is used that estimates it from the character height. Usually, writing has a fixed writing direction. For example, in horizontal writing, writing is always added horizontally within a line. The description analysis unit 11 uses this characteristic to detect the writing direction. Then, using a direction perpendicular to the writing direction as an axis, it calculates the maximum value of the height of a group of consecutive strokes as the size of the characters belonging to that group. Figure 4 shows a schematic diagram of how the description analysis unit 11 detects character size. In Figure 4, the writing direction is horizontal, so the axis perpendicular to that direction is vertical. The tallest character in the group of consecutive vertical strokes is the character "a." Therefore, the size of the characters included in this writing is calculated from the height of "a."

[0036] <Grouping process for projection onto information sharing board> The grouping process for projection onto the information sharing board is performed in response to an image being projected onto the information sharing board from an information device such as a projector. For example, closely related descriptions are grouped together in advance, one page at a time, and the information device such as a projector switches the image of the projected page by turning each page, thereby switching the image of the page projected onto the information sharing board. Therefore, the projection area corresponding to the entire page is treated as one group. For simplicity, a change in projection content is detected as a new projection and grouping process is performed. Note that the projection area is a simple rectangle and is brighter than the surrounding area, so it can be easily detected using known image processing techniques. Similarly, a change in projection content due to turning pages can also be detected using known image processing techniques.

[0037] The grouping process for projection onto the information sharing board will be described with reference to the flowchart in Fig. 5. In step S501, the projection detection unit 12 detects, from frames captured by the imaging device 102, the projection of an image of a page onto the information sharing board and a change in the projection content due to page turning.

[0038] In step S502, the projection analysis unit 13 detects the projection area onto which the image of the page is projected on the information sharing board from the frame captured by the imaging device 102, and determines whether the detected projection area overlaps with the group area of ​​an existing group.

[0039] If it is determined that there is overlap as a result of this determination, the process proceeds to step S505, and if it is determined that there is no overlap, the process proceeds to step S503. In determining whether there is overlap, even if the entire projection area does not overlap, if the area of ​​the projection area overlaps by a percentage equal to or greater than a threshold, it may be determined that there is overlap. Here, as described above, overlap occurs when the projection content is turned to the next page.

[0040] In step S505, the projection analysis unit 13 identifies a group among the existing groups that is determined to overlap with the projection area. Then, when the process proceeds from step S505 to step S504, in step S504, the description generation unit 18 updates the "group position" included in the description information of the group identified in step S505 so that the projection area is included in the group area represented by the "group position."

[0041] On the other hand, in step S503, the projection analysis unit 13 creates a new group for the projection area detected in step S502. When the process proceeds from step S503 to step S504, in step S504 the description generation unit 18 generates description information for the group created in step S503 based on the projection area detected in step S502.

[0042] The description information generated for each group by the description generating unit 18 in step S504 includes the position of the group, the allowable zoom range for the group, and the time for the group, similar to the description information generated in step S304 above.

[0043] The position of a group is position information that defines a rectangle (group area) that surrounds the projection area included in the group. The information about the rectangle represented by the position information is the same as the position of the group related to the description.

[0044] The allowable zoom range for a group is a constraint on the zoom range when including the group in the field of view, and is calculated in the same way as for description groups. That is, if the projection area includes text, it is calculated based on the size of the text, and otherwise a default value is used. The time for a group is a logical time, and is information that can identify the order of generation / update between groups.

[0045] <Time update process> The time update process is performed when the lecturer points to a description or projection area that was previously written for the purpose of explanation. The time update process will be described with reference to the flowchart in FIG.

[0046] In step S601, the instruction detection unit 14 detects the movement performed by the instructor using frames captured by the imaging device 102. There are various detection methods for detecting the movement performed by the instructor.

[0047] For example, when the instruction detection unit 14 detects a group area in a frame where the distance between the instructor's hand and the group area is below a threshold, it determines that the action of "the instructor has pointed with his hand to the group corresponding to the group area" has occurred.

[0048] Furthermore, for example, when the instruction detection unit 14 detects a group area where the distance between the position in the frame and the target of the laser pointer operated by the instructor is below a threshold, it determines that the action of "the instructor pointed to the group corresponding to the group area with the laser pointer" has been performed. The same applies when the instructor points to a projection area using the laser pointer. When the target pointed to by the instructor using the laser pointer is a projection area, the instruction may also be given by operating an information processing device using a mouse pointer or the like. In this case, the operation of the information processing device by the instructor is detected and treated as an instruction for the projection area. In order to improve the accuracy of detection, the instructor's remarks may be recognized and their content may be added to the determination criteria.

[0049] In step S602, the instruction analysis unit 15 identifies the group determined to have been pointed out by the instructor in the process of step S601. In step S603, the description generation unit 18 updates the time included in the description information of the group identified in step S602. When the time included in the description information is updated, for example, if real time is used as the time, the time is updated to the current time (instructed time).

[0050] <Target position recording process, target position update process> The target position recording process and target position updating process for an information sharing board are processes for efficiently updating the position of writing or projection when the blackboard serving as the information sharing board can slide up and down or when the whiteboard serving as the information sharing board can be moved. First, the target position recording process records the position of the information sharing board at the start of imaging by the imaging device 102. Then, the target position updating process updates the recorded position of the information sharing board in response to changes in the position of the information sharing board during imaging by the imaging device 102.

[0051] 7, the target position recording process for recording the initial position of the information sharing board (the position of the information sharing board at the time when the imaging device 102 starts capturing images) in the storage unit 202 will be described. In step S701, the target detection unit 16 detects the information sharing board from the frames captured by the imaging device 102.

[0052] In step S702, the object analysis unit 17 detects the position of the information sharing board detected in step S701. For example, the object analysis unit 17 converts the position of the information sharing board in the frame detected in step S701 (the positions of the four corners of the information sharing board, the positions of two corresponding corners on the information sharing board, etc.) into a position in a world coordinate system within the lecture room. The world coordinate system within the lecture room is a coordinate system with a single point within the lecture room as the origin and three axes orthogonal to each other at the origin as the x-axis, y-axis, and z-axis, respectively. In this embodiment, the position and orientation of the image capture device 102 in the world coordinate system within the lecture room are assumed to be acquired in advance as position and orientation information and stored in the storage unit 202. Therefore, the object analysis unit 17 converts the position of the information sharing board in the frame captured by the image capture device 102 by known coordinate transformation using the position and orientation information, and acquires this position as the "position in the world coordinate system within the lecture room."

[0053] In step S703, the description generation unit 18 generates position information that defines a rectangle that surrounds the area of ​​the information sharing board defined by the position obtained in step S702. The position information that defines the rectangle may be the positions of the four corners of the rectangle, the positions of the upper left corner and lower right corner of the rectangle, or the position of the upper left corner of the rectangle and the height and width of the rectangle. The description generation unit 18 then stores the generated position information in the storage unit 202 as the "initial position of the information sharing board."

[0054] Next, the target position update process for updating the position of the information sharing board will be described with reference to the flowchart in Fig. 8. In step S801, the target detection unit 16 detects movement of the information sharing board from frames captured by the imaging device 102. In step S802, the target analysis unit 17 identifies which information sharing board has moved.

[0055] In step S803, the description recording unit 18 determines, as described above, the current position in the world coordinate system of the information sharing board (the information sharing board identified in step S802) detected from the frame captured by the imaging device 102. In step S804, the description recording unit 18 stores in the memory unit 202 the position in the world coordinate system of the "moved information sharing board" determined in step S803.

[0056] An example of the configuration of description information is shown in Fig. 9. Fig. 9 shows the configuration of description information for each group, with each line corresponding to the description information for the corresponding group. In Fig. 9, the description information includes the position of the group, the allowable zoom range for the group, the time for the group, the description target, and the description target position.

[0057] 9 is position information indicating the positions of the upper left and lower right corners of a rectangle that surrounds the descriptions included in the group in a local coordinate system based on the information sharing board. The local coordinate system based on the information sharing board is, for example, a coordinate system in which a point on the information sharing board is the origin, an axis along one side of the information sharing board is the x-axis, and an axis along an orthogonal side perpendicular to the one side of the information sharing board is the y-axis.

[0058] The allowable zoom range for a group is a constraint on the zoom range when including the group in the field of view. The allowable zoom range for a group that includes text is the upper and lower limits of the zoom range at which the viewer of the viewing terminal 104 can read the text, and the allowable zoom range for a group that does not include text is a specified zoom ratio. The time for a group is a logical time as described above. The object to be described is a unique identifier for each information sharing board.

[0059] The positions to be described are the positions of the upper left corner and the lower right corner of the information sharing board in the world coordinate system in the lecture room. It would be more efficient to manage the positions to be described separately, but for convenience of explanation, they are listed together here. The structure of the description information is not limited to a specific structure, and for example, the information group shown in FIG. 9 may be managed by dividing it into two groups. Furthermore, the device that stores each piece of information described above is not limited to a specific device, and it is sufficient if the imaging control device 101 can acquire it as needed.

[0060] FIG. 10(a) shows the content represented by the description information of the group (referred to as group A) corresponding to the first line in FIG. 9. The positions of the upper left corner and lower right corner of the information sharing board (information sharing board b1) assigned identifier b1 in the world coordinate system within the lecture room are (gx1s, gy1s) and (gx1e, gy1e), respectively, and group A is on information sharing board b1. Group A includes the character string "abc def ghi" as a description, and the positions of the upper left corner and lower right corner of the rectangle surrounding the character string are (x1s, y1s) and (x1e, y1e), respectively. The latest creation time / update time / instruction time for group A is t1. The upper limit of the zoom range for the character strings belonging to group A is 1.5, and the lower limit is 0.5. Figure 10(b) shows the content represented by the description information of the group (referred to as group B) corresponding to the second line in Figure 9. The positions of the upper left corner and lower right corner of the information sharing board (information sharing board b2) assigned identifier b2 in the world coordinate system within the lecture room are (gx2s, gy2s) and (gx2e, gy2e), respectively, and group B is on information sharing board b2. Group B includes a line drawing as a description, and the positions of the upper left corner and lower right corner of the rectangle that surrounds the line drawing are (x2s, y2s) and (x2e, y2e), respectively. The latest generation time / update time / instruction time for this group B is t2. The zoom range for the line drawing belonging to this group B is 1.

[0061] <Post-processing> Next, the subsequent processing will be described. The subsequent processing can include a process for determining the angle of view and a synthesis process.

[0062] <Angle of view determination process> The field of view determination process according to this embodiment will be described with reference to the flowchart in Fig. 11. In step S1101, the field of view determination unit 19 references the description information of each group, and acquires the description information including the latest time from the description information of groups not included in the currently set field of view (set field of view) in frames captured by the imaging device 102. The initial value of the set field of view is an appropriate initial value, and depending on the initial value, no groups may be included.

[0063] In step S1102, the angle-of-view determination unit 19 updates the set angle of view so that the group corresponding to the description information acquired in step S1101 is included in the set angle of view. That is, the angle-of-view determination unit 19 updates the set angle of view so that the position of the group included in the description information acquired in step S1101 is included in the set angle of view. That is, in this embodiment, the group with the most recent time is given priority for inclusion in the set angle of view.

[0064] In step S1103, the angle-of-view determination unit 19 determines whether the zoom ratio according to the set angle of view is included in the allowable zoom range included in the description information of each group included in the set angle of view.

[0065] If the result of this determination is that the zoom ratio corresponding to the set angle of view is included in the allowable zoom range included in the description information of each group included in the set angle of view, the process proceeds to step S1101. On the other hand, if the zoom ratio corresponding to the set angle of view is not included in the allowable zoom range included in the description information of one or more groups included in the set angle of view, the process proceeds to step S1104. In step S1104, the angle of view determination unit 19 returns (restores) the set angle of view to the set angle of view before updating in step S1102.

[0066] 11 from a frame captured by the imaging device 102, and distributes the thus-cut image as a frame for distribution to the viewing terminal 104. The image cut out from the frame may be resized to an appropriate size (for example, the size of a frame for distribution) as necessary, and then distributed to the viewing terminal 104 as a frame for distribution.

[0067] <Angle of view synthesis processing> The angle of view synthesis process is a process for creating a frame for distribution when closely related descriptions are far apart and cannot both be contained in a single angle of view. A new angle of view is set for the description, and multiple angles of view including the set angle of view are synthesized to create a frame for distribution. At this time, the same angle of view determination process as described above is performed for each of the angles of view that have already been synthesized. The angle of view synthesis process according to this embodiment will be described with reference to the flowchart in FIG. 12.

[0068] In step S1201, the field of view determination unit 19 refers to the description information of each group and acquires the description information including the latest time from the description information of the group that is not yet included in the synthesized field of view in the frame captured by the imaging device 102.

[0069] In step S1202, the angle of view determination unit 19 changes the synthesized angle of view (all or part of the synthesized angle of view) so that the synthesized angle of view and the angle of view of the group of descriptive information acquired in step S1201 fit within the angle of view for transmission (e.g., a frame for distribution).

[0070] In step S1203, the angle of view determination unit 19 determines whether or not one or more of the changed composite angles of view have a "zoom ratio corresponding to the changed angle of view within the zoom allowable range." If the result of this determination is that one or more of the changed composite angles of view have a "zoom ratio corresponding to the changed angle of view within the zoom allowable range," the process proceeds to step S1208. On the other hand, if none of the changed composite angles of view have a "zoom ratio corresponding to the changed angle of view within the zoom allowable range," the process proceeds to step S1204.

[0071] In step S1208, the angle of view determination unit 19 selects, from the changed combined angles of view, an angle of view for which the "zoom factor corresponding to the changed angle of view is within the zoom tolerance range" as a selected angle of view.The angle of view determination unit 19 then restores the changed combined angles of view other than the selected angle of view to their pre-change state.As a result, the angles of view of the group of descriptive information acquired in step S1201 are included in the combined angles of view.On the other hand, in step S1204, the angle of view determination unit 19 restores the angle of view changed in the most recent step S1202 to its original angle of view.

[0072] In step S1205, the angle of view synthesis unit 20 generates a frame for distribution in which partial images within each angle of view included in the synthesized angle of view are arranged so that the positional relationship of the partial images within each angle of view is maintained in at least one of the horizontal and vertical directions.

[0073] For example, as shown on the left side of Fig. 13(A), assume that partial image X is located above and to the left of partial image Y in a frame captured by the imaging device 102. In this case, as shown on the right side of Fig. 13(A), a frame for distribution is generated in which partial image X and partial image Y are arranged side by side so that the vertical positional relationship between partial image X and partial image Y is maintained.

[0074] 13(B), for example, assume that in a frame captured by the imaging device 102, partial image U is located to the lower left of partial image V and to the upper left of partial image W, and partial image V is located to the upper right of partial image W. In this case, as shown on the right side of Fig. 13(B), a frame for distribution is generated in which partial images U, V, and W are arranged so that the horizontal positional relationship between partial image U and partial image V is maintained, and so that the vertical positional relationship between partial image U / V and partial image W is maintained.

[0075] In step S1206, the angle of view synthesis unit 20 determines whether or not the frame for distribution generated in step S1205 includes, without loss, image portions within each of the angles of view included in the synthesized angle of view.

[0076] If it is determined that the frame for distribution generated in step S1205 contains all of the image portions within the respective angles of view included in the combined angle of view without any loss, the process proceeds to step S1201.

[0077] On the other hand, if it is not determined that the frame for distribution generated in step S1205 includes all of the image portions within the respective angles of view included in the combined angle of view without any loss, the process proceeds to step S1207.

[0078] In step S1207, the field of view synthesis unit 20 generates the frame for distribution generated in the previous step S1205, i.e., the frame for distribution before placing the last placed ``image portion of the field of view of the group of descriptive information acquired in step S1201.''

[0079] In this way, according to this embodiment, the information written on the information sharing board can be grouped, and the angle of view for simultaneously distributing closely related information can be determined according to the priority based on the group's creation time / update time / instruction time, etc.

[0080] [Second embodiment] In each of the following embodiments, including this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, they will be considered to be the same as the first embodiment. In this embodiment, a case will be described in which the imaging device 102 has a camera (PTZ camera) capable of controlling pan, tilt, and zoom. When a PTZ camera is used, the angle of view to be controlled is the imaging angle of view of the PTZ camera itself, and is changed by controlling pan, tilt, and zoom. Furthermore, the zoom ratio of the angle of view corresponds to the zoom ratio of the PTZ camera.

[0081] In this embodiment, the angle of view determined by the processing according to the first embodiment is notified to the imaging device 102. In the first embodiment, the change of the angle of view is realized only by the imaging control device 101, whereas in this embodiment, it is necessary to control the PTZ camera of the imaging device 102. Therefore, when the angle of view determination unit 19 determines a new angle of view, it outputs the determined angle of view to the imaging device 102 via the interface 203 as a setting related to pan-tilt-zoom.

[0082] It is not necessary to notify the image capture device 102 of the candidate angles of view generated during the angle of view determination process each time. The image capture control device 101 incorporates a logical model of the PTZ camera, making it possible to calculate the zoom ratio after changing the angle of view through simulation. Therefore, it is possible to determine in step S1103 whether the zoom ratio falls within the allowable zoom range without changing the actual angle of view of the camera.

[0083] As described above, this embodiment can be applied to an image capturing device 102 having one or more PTZ cameras, thereby achieving the same effects as those of the first embodiment.

[0084] [Third embodiment] In the configuration of Figure 1, the imaging device 102 and the imaging control device 101 are described as separate devices, but the imaging device 102 and the imaging control device 101 may be integrated, for example, the imaging device 102 may have the imaging control device 101.

[0085] 1, the imaging control device 101 and the viewing terminal 104 are described as being connected via the network 103. However, for the purpose of ensuring scalability, a distribution server may be provided on the network 103, and the server may distribute distribution frames generated by the imaging control device 101 to the viewing terminal 104.

[0086] In the second embodiment, video streams from multiple PTZ cameras are used without processing, but if the frames captured by the PTZ cameras have a sufficiently high resolution, the method of the first embodiment may be combined. In other words, video clips extracted from the video streams from the PTZ cameras may be combined with the video being distributed.

[0087] Furthermore, some of the above-described processes that are described as being performed by the imaging control device 101 may be executed by the viewing terminal 104. For example, if there is variation in the screen resolution or size of the viewing terminal 104, the above-described field of view determination process may be performed by the viewing terminal 104, and the allowable zoom range may be adjusted to suit the viewing terminal 104. Furthermore, if only viewing terminals 104 with a certain level of resolution are allowed to allow participants to select a field of view, the viewing terminal 104 may repeat the field of view determination process to generate multiple fields of view to use as selection candidates.

[0088] Furthermore, the numerical values, processing timing, processing order, processing subject, destination / source / storage location of data (information) used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0089] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0090] (Other embodiments) The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0091] The disclosure of this specification includes the following imaging control device, imaging control method, and computer program.

[0092] (Item 1) a grouping means for determining continuity between a description added to the imaging target and a description added to the imaging target earlier than the description added to the imaging target from a frame obtained by imaging the imaging target, and for grouping the descriptions in the imaging target based on the result of the determination; a control means for controlling the angle of view of an imaging device so as to capture a description of the imaging target of the group to be distributed in the frame, based on the time related to each group obtained by the grouping means; An imaging control device comprising:

[0093] (Item 2) The imaging control device described in item 1 is characterized in that the grouping means determines the spatial and / or temporal continuity between a description added to the imaging object and a description added to the imaging object earlier than the description added to the imaging object.

[0094] (Item 3) The imaging control device described in item 1 or 2 is characterized in that, when the grouping means determines that there is continuity between a description added to the imaging target and a description that was added to the imaging target earlier than the description added to the imaging target, the grouping means includes the description added to the imaging target in a group of descriptions that was added to the imaging target earlier than the description added to the imaging target.

[0095] (Item 4) The imaging control device described in any one of items 1 to 3 is characterized in that, when it is determined that there is no continuity between a description added to the imaging target and a description that was added to the imaging target earlier than the description added to the imaging target, a group of descriptions added to the imaging target is generated.

[0096] (Item 5) The imaging control device according to any one of items 1 to 4, characterized in that the grouping means determines whether or not there is an overlap between the projection area of ​​the page projected onto the imaging target and the area of ​​an existing group from a frame capturing an image of the imaging target, and groups the projection area of ​​the imaging target based on the result of the determination.

[0097] (Item 6) Item 6. The imaging control device according to item 5, characterized in that, when the grouping means determines that there is an overlap between the projection area of ​​the page projected onto the imaging target and the area of ​​the existing group, the grouping means includes the projection area of ​​the page projected onto the imaging target in the existing group.

[0098] (Item 7) 7. The imaging control device according to item 5 or 6, characterized in that, when it is determined that there is no overlap between the projection area of ​​the page projected onto the imaging target and the area of ​​the existing group, the grouping means generates a group of the projection area of ​​the page projected onto the imaging target.

[0099] (Item 8) moreover, 8. The imaging control device according to any one of items 1 to 7, further comprising means for detecting movement of the imaging target and acquiring the position of the imaging target.

[0100] (Item 9) The imaging control device according to item 1, characterized in that the control means updates the set angle of view to include the groups in order starting with the group with the most recent time, and sets the set angle of view after the update as the angle of view until the zoom ratio corresponding to the set angle of view is no longer included in the allowable zoom range of each group included in the set angle of view.

[0101] (Item 10) 10. The imaging control device according to any one of items 1 to 9, wherein the control means determines the allowable zoom range of a group including characters in accordance with the size of the characters.

[0102] (Item 11) moreover, The imaging control device described in item 1 is characterized in that it comprises a means for generating, as a frame for distribution, a frame in which partial images within each of the angles of view included in the angle of view are arranged so that the positional relationship of the partial images within each of the angles of view is maintained in at least one of the horizontal and vertical directions.

[0103] (Item 12) moreover, 11. The imaging control device according to any one of items 1 to 10, further comprising means for outputting the determined angle of view to an imaging device having a camera capable of controlling pan, tilt, and zoom.

[0104] (Item 13) 13. The imaging control device according to any one of items 1 to 12, wherein the time relating to the group includes a generation time of the group, an update time for the group, and an instruction time for the group.

[0105] (Item 14) 14. The imaging control device according to any one of items 1 to 13, wherein the imaging target includes a blackboard and a screen.

[0106] (Item 15) An imaging control method performed by an imaging control device, a grouping step in which a grouping means of the imaging control device determines continuity between a description added to the imaging target and a description added to the imaging target earlier than the description added to the imaging target from a frame in which the imaging target is imaged, and groups the descriptions in the imaging target based on the result of the determination; a control step in which a control means of the imaging control device controls an angle of view of the imaging device so as to capture a description of the imaging target of the group to be distributed in the frame, based on the time related to each group obtained in the grouping step; An imaging control method comprising:

[0107] (Item 16) A computer program for causing a computer to function as each of the means of the imaging control device according to any one of items 1 to 14.

[0108] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present disclosure. Accordingly, the following claims are appended to apprise the public of the scope of the present disclosure. [Explanation of symbols]

[0109] 10: Description detection unit 11: Description analysis unit 12: Projection detection unit 13: Projection analysis unit 14: Instruction detection unit 15: Instruction analysis unit 16: Object detection unit 17: Object analysis unit 18: Description generation unit 19: View angle determination unit 20: View angle synthesis unit

Claims

1. a grouping means for determining continuity between a first description added to an imaging target and a second description added to the imaging target earlier than the first description, based on a frame obtained by imaging the imaging target, and for grouping descriptions in the imaging target based on a result of the determination; a control means for controlling the angle of view of an imaging device so as to capture a description of the imaging subject of the group to be distributed in the frame, based on the time related to each group obtained by the grouping means; An imaging control device comprising:

2. The imaging control device according to claim 1, characterized in that the grouping means determines spatial and / or temporal continuity between a first description added to the imaging object and a second description added to the imaging object earlier than the first description.

3. The imaging control device according to claim 1, characterized in that when the grouping means determines that there is continuity between a first description added to the imaging target and a second description that was added to the imaging target earlier than the first description, the grouping means includes the first description added to the imaging target in a group of the second description that was added to the imaging target earlier than the first description.

4. The imaging control device according to claim 1, characterized in that when the grouping means determines that there is no continuity between a first description added to the imaging target and a second description that has been added to the imaging target before the first description, it generates a group of the first description added to the imaging target.

5. The imaging control device according to claim 1, characterized in that the grouping means determines whether or not there is an overlap between a projection area of ​​a page projected onto the imaging target from a frame capturing an image of the imaging target and an area of ​​an existing group, and groups the projection areas of the imaging target based on the result of the determination.

6. The imaging control device according to claim 5, characterized in that when the grouping means determines that there is an overlap between the projection area of ​​the page projected onto the imaging target and the area of ​​the existing group, the grouping means includes the projection area of ​​the page projected onto the imaging target in the existing group.

7. The imaging control device according to claim 5, characterized in that when the grouping means determines that there is no overlap between the projection area of ​​the page projected onto the imaging target and the area of ​​the existing group, it generates a group of the projection area of ​​the page projected onto the imaging target.

8. moreover, 2. The imaging control device according to claim 1, further comprising: means for detecting a movement of the imaging target and acquiring a position of the imaging target.

9. 2. The imaging control device according to claim 1, wherein the control means updates the set angle of view to include the groups in order starting with the group with the most recent time, and sets the set angle of view to the angle of view after the update until a zoom ratio corresponding to the set angle of view is no longer included in an allowable zoom range of each group included in the set angle of view.

10. 2. The image capture control device according to claim 1, wherein the control means determines the allowable zoom range for a group including characters in accordance with a size of the characters.

11. moreover, The imaging control device according to claim 1, further comprising a means for generating a frame for distribution in which partial images within each of the angles of view included in the angle of view are arranged so that the positional relationship of the partial images within each of the angles of view is maintained in at least one of the horizontal and vertical directions.

12. moreover, 2. The imaging control device according to claim 1, further comprising: means for outputting the determined angle of view to an imaging device having a camera capable of controlling pan, tilt and zoom.

13. 2. The imaging control device according to claim 1, wherein the time relating to the group includes a generation time of the group, an update time for the group, and an instruction time for the group.

14. 2. The imaging control device according to claim 1, wherein the imaging target includes a blackboard and a screen.

15. An imaging control method performed by an imaging control device, a grouping step in which a grouping means of the imaging control device judges continuity between a first description added to the imaging target and a second description added to the imaging target earlier than the first description, based on a frame in which the imaging target is imaged, and groups the descriptions in the imaging target based on a result of the judgment; a control step in which a control means of the imaging control device controls an angle of view of the imaging device so as to capture a description of the imaging subject of the group to be distributed in the frame, based on the time related to each group obtained in the grouping step; An imaging control method comprising:

16. A computer program for causing a computer to function as each of the means of the imaging control device according to any one of claims 1 to 14.

Images