Processing apparatus, method for controlling processing apparatus, and program
The processing device enhances video clarity by adjusting transparency based on human body actions, ensuring superimposed information is visible during explanations.
Patent Information
- Application Number
- JP2023073644
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-27
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-04-27
AI Technical Summary
Existing technologies fail to dynamically adjust the transparency of superimposed information based on the human body's actions, making it difficult to visually recognize the information when superimposed on a video.
A processing device that extracts the human body region, determines its actions, and adjusts the transparency of superimposed information based on the detected motion or audio cues, ensuring clarity by making overlapping regions transparent when the body is explaining.
Enhances the visibility of superimposed information by making it clearer when the human body is performing an explanation motion, allowing viewers to understand the content more effectively.
Smart Images

Figure 2024158439000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a processing device, a control method for the processing device, and a program. [Background technology]
[0002] In the past, when automatically recording lecture scenes, it has become common to overlay information that the human body is explaining on the video of the human body. In such cases, efforts are made to ensure that the back of the overlaid video is not difficult to see.
[0003] In Patent Document 1, when a second image (CG person or sign language interpreter) is superimposed on a first image (background), the display position and transparency of the second image are controlled using image information extracted from the first image, making the background easier to see. The image information extracted from the first image is a saliency map created from program information, and an area that people pay attention to. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent No. 6046961 Summary of the Invention [Problem to be solved by the invention]
[0005] However, in Patent Document 1, when the second image is superimposed on the first image, the transparency of the area overlapping with the human body area cannot be changed depending on whether the human body is making an explanatory movement or not. Furthermore, when information (the second image in Patent Document 1) is superimposed on an image of a human body (the first image in Patent Document 1), it may be difficult to visually recognize the information in the superimposed image. Therefore, there is a concern that viewers will not be able to understand the content that the human body is explaining through the superimposed information.
[0006] Therefore, an object of the present invention is to provide a processing device that can make information more easily visible when the information is superimposed on a video image. [Means for solving the problem]
[0007] In order to achieve the above-mentioned object, a processing device as one aspect of the present invention is characterized in having a first extraction means for extracting a region of a human body in an image, a superimposition means for superimposing predetermined superimposition information on the image, a first judgment means for judging whether the human body is performing a predetermined movement and outputting a judgment result, a second extraction means for extracting a region of the predetermined superimposition information that overlaps with the region of the human body as an overlapping region based on the judgment result, the region of the human body, and the predetermined superimposition information, and a transparency change means for changing the transparency of at least a portion of the predetermined superimposition information to be higher in accordance with the overlapping region. Effect of the Invention
[0008] According to the present invention, it is possible to provide a processing device that can make information more easily visible when the information is superimposed on a video image. [Brief description of the drawings]
[0009] [Figure 1] 1 is a block diagram showing the configuration of an automatic photography system according to a first embodiment. [Diagram 2] 1 is a block diagram showing the configuration of an image superimposing device according to a first embodiment. [Diagram 3] 4A to 4C are diagrams illustrating a result of human body skeleton estimation according to the first embodiment. [Figure 4] 1A to 1C are diagrams illustrating a method for determining explanatory movements of a human body using a skeleton estimation result of the human body according to the first embodiment. [Diagram 5] 13A to 13C are diagrams illustrating a state in which explanatory material is superimposed on a human body image captured when the human body is not performing an explanatory motion in the first embodiment. [Figure 6]13A to 13C are diagrams illustrating a state in which explanatory material in which an area overlapping with a human body area has been made transparent is superimposed on a human body photographed video when the human body is making an explanatory movement in the first embodiment. [Figure 7] FIG. 13 is a diagram illustrating a state in which explanatory material is superimposed on a human body image captured when the human body is making an explanatory movement, with the area overlapping with the arm of the human body being made transparent in the image in accordance with embodiment 1. [Figure 8] 13A to 13C are diagrams illustrating a state in which explanatory material is superimposed on a captured image of a human body when the human body is making an explanatory movement in accordance with the first embodiment, with the area overlapping with the face area of the human body being made transparent. [Figure 9] 13A to 13C are diagrams illustrating a state in which explanatory material is superimposed on a human body image captured when the human body is not performing an explanatory motion in the first embodiment. [Figure 10] 4 is a flowchart showing a processing procedure of the automatic photography system according to the first embodiment. [Figure 11] FIG. 11 is a block diagram showing the configuration of an automatic photography system according to a second embodiment. [Figure 12] 13A and 13B are diagrams illustrating a state in which explanatory material in which an area overlapping with a human body area is made transparent is superimposed on a captured image of the human body when the human body is speaking in the second embodiment. [Figure 13] 13 is a diagram showing a speech section of a human body and a transparent section of a material area according to the second embodiment. FIG. [Figure 14] 10 is a flowchart showing a processing procedure of an automatic photography system according to a second embodiment. [Figure 15] FIG. 11 is a block diagram showing the configuration of an automatic photography system according to a third embodiment. [Figure 16] FIG. 13 is an explanatory diagram for identifying an explanatory area based on the contents of an utterance and the contents of an explanatory material in the third embodiment. [Figure 17] 11 is a flowchart showing a processing procedure of an automatic photography system according to a third embodiment. [Figure 18] 11 is a flowchart showing a processing procedure of an automatic photography system according to a third embodiment. [Figure 19] FIG. 11 is a block diagram showing the configuration of an automatic photography system according to a fourth embodiment. [Figure 20] 13 is a diagram showing how an explanation region is specified based on an explanation action of a human body according to the fourth embodiment. FIG. [Figure 21] This figure explains the state in which explanatory material is superimposed on a human body image in which the area overlapping the human body area has been made transparent when the human body makes an explanatory motion in embodiment 4 and the human body and the explanatory area do not overlap. [Figure 22] A figure explaining the state in which explanatory material that has been subjected to emphasis processing is superimposed on the explanation area on the human body photographed video in the case where the human body and the explanation area overlap when the human body performs an explanation motion in embodiment 4. [Diagram 23] A figure explaining the state in which explanatory material that has been subjected to emphasis processing is superimposed on the explanation area on the human body photographed image in the case where the human body and the explanation area overlap when the human body gives a spoken explanation in embodiment 4. [Figure 24] 13 is a flowchart showing a processing procedure of an automatic photography system according to a fourth embodiment. [Diagram 25] 13 is a flowchart showing a processing procedure of an automatic photography system according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] The following describes in detail the embodiments for carrying out the present invention. Note that the embodiments described below are examples for realizing the present invention, and should be appropriately modified or adjusted depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments. Also, in all drawings, parts having the same functions are given the same numbers, and repeated explanations are omitted.
[0011] <Embodiment 1> An example of the configuration of the image superimposing device A1003 according to the first embodiment will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of an automatic photography system A1000 including the image superimposing device A1003 according to the first embodiment. The image superimposing device A1003 functions as a processing device (information processing device) that performs various processes such as human body extraction processing, explanatory action determination processing, and image superimposing processing using various functional units to be described later.
[0012] The automatic photography system A1000 detects a human body (person) from a photographed image (image information) and determines the explanatory action of the detected human body.Then, it makes transparent (changes the transparency) an area (overlapping area) that overlaps with the area of the human body performing the explanatory action in the acquired explanatory material (superimposed information superimposed on the image).Then, it superimposes the explanatory material with the changed transparency on the image information (photographed image of the human body) and displays the superimposed image on a monitor.
[0013] The automatic photography system A1000 is configured to include an image capturing device A1001, a material capturing device A1002, an image superimposing device A1003, and a monitor device A1013. The image superimposing device A1003 is communicably connected to the image capturing device A1001, the material capturing device A1002, and the monitor device A1013. The image superimposing device A1003 and the monitor device A1013 are also connected via a line such as a video interface.
[0014] The image acquisition device A1001 is a device that acquires images by photographing the surroundings of the image acquisition device A1001 and generates a captured image from the multiple captured images, and is composed of an imaging device such as a camera. The image acquisition device A1001 has an imaging unit (not shown), which is composed of a lens unit for forming an image of light, an imaging element that converts the imaged light into an analog signal, and a signal processing unit. The imaging unit also functions as a photographing means that photographs a subject and acquires an image. The image acquisition device A1001 outputs video information generated from the multiple captured images to the image superimposition device A1003.
[0015] The material acquisition device A1002 is a device that acquires explanatory materials such as presentation materials created by Microsoft PowerPoint or Adobe PDF in electronic data. The material acquisition device A1002 outputs the acquired explanatory materials to the image superimposition device A1003. The explanatory materials may be any images or information to be superimposed on a video. The explanatory materials may be, for example, text information. In other words, the explanatory materials may be predetermined superimposition information to be superimposed on a video. Here, the predetermined superimposition information may be an image or text, or may be other symbols, icons, or the like. The explanatory materials are used, for example, by a human body included in the video. The human body included in the video can explain the contents of the predetermined superimposition information while watching the video on which the predetermined superimposition information is superimposed through a monitor or the like.
[0016] The image superimposing device A1003 detects a human body from the image input from the image capturing device A1001 and judges whether the human body is making an explanation motion. If the human body is making an explanation motion, the image superimposing device A1003 makes the area of the explanation material that overlaps with the human body transparent and superimposes it on the image information. The image superimposed is then output to the monitor device A1013.
[0017] The image superimposing device A1003 is configured to have, as its functional units, a video acquisition unit A1004, a document acquisition unit A1005, a skeletal information estimation unit A1006, a human body movement determination unit A1007, a region division processing unit A1008, and an overlap region extraction unit A1009. Furthermore, the image superimposing device A1003 is configured to have, as its functional units, a transparency change unit A1010, an image superimposing unit A1011, and a video output device A1012. Each of these functional units is realized by the CPU 11, which will be described later, expanding a program stored in the ROM 12 into the RAM 13 and executing the program. The CPU 11 then stores the execution results of each process, which will be described later, in the RAM 13 or a predetermined storage medium.
[0018] The video acquisition unit A1004 acquires video information. Specifically, the video acquisition unit A1004 acquires video information input from the video acquisition device A1001. However, this is not limiting, and the video acquisition unit A1004 may acquire video information from a device or a server other than the video acquisition device A1001. The video acquisition unit A1004 outputs the acquired video information to the skeleton information estimation unit A1006 and the region division processing unit A1008.
[0019] The skeletal information estimation unit A1006 estimates skeletal information of the human body. Specifically, the skeletal information estimation unit A1006 detects a human body from the video information input from the video acquisition unit A1004, and estimates skeletal information, which is information on the skeleton of the human body. The skeletal information estimation unit A1006 detects a human body from an image included in the video information (based on the image), and estimates skeletal information of the detected human body. When estimating the skeletal information of the human body, the skeletal information estimation unit A1006 cuts out the coordinates of the human body in the video information, and estimates the skeletal information of the human body using a skeleton estimation technique. Thereafter, the skeletal information estimation unit A1006 outputs the video information and the estimated skeletal information of the human body to the human body movement determination unit A1007 as a skeleton estimation result. In this embodiment, the skeletal information estimation unit A1006 also functions as an estimation unit when estimating the skeleton of the detected human body and outputting the skeleton estimation result.
[0020] In recent years, many skeletal estimation technologies using Deep Learning have been released, and it has become possible to estimate the human body skeleton with high accuracy. Among them, some technologies such as OpenPose and DeepPose are provided as OSS (Open Source Software), making it easier to perform skeletal estimation. In the first embodiment, although there is no limitation on the skeletal estimation technology, any of the above-mentioned skeletal estimation technologies using Deep Learning is used.
[0021] The human body motion determination unit A1007 determines whether the human body is performing a predetermined motion. Specifically, it determines whether the human body is performing an explanatory motion as a predetermined motion using human body skeletal information, which is an estimation result obtained from the skeletal information estimation unit A1006. If the human body motion determination unit A1007 determines that the human body has performed an explanatory motion, it outputs the determination result, video information, and skeletal estimation result to the overlapping area extraction unit A1009. On the other hand, if it determines that the human body has not performed an explanatory motion, it outputs the determination result and video information to the image superimposition unit A1011. The determination process performed by the human body motion determination unit A1007 will be described below with reference to FIG. 3 and FIG. 4. In this embodiment, the human body motion determination unit A1007 also functions as a first determination means for determining whether the human body is performing a predetermined motion and outputting the determination result.
[0022] FIG. 3 is a diagram showing skeletal information of the shoulder, arm, and neck required to determine the explanatory motion from the human body skeletal estimation result acquired from the human body motion determination unit A1007. D001 represents video information. P001 represents the human body. P002 represents the left hand, P003 represents the left elbow, and P004 represents the left shoulder. P005 represents the neck. P006 represents the right shoulder, P007 represents the right elbow, and P008 represents the right hand.
[0023] FIG. 4 is a diagram for explaining a state when determining an explanatory motion of a right arm of a human body. D101 represents video information. P101 represents a human body. P106 represents a right shoulder, P107 represents a right elbow, and P108 represents a right hand. The angle between the right shoulder P106 and the right elbow P107 is P109. The angle between the right elbow P107 and the right hand P108 is P110.
[0024] The human body motion determination unit A1007 can determine that an explanation motion is being performed when, for example, P108 and P109 are 0° or more and less than 90°. Note that this is merely an example, and any method may be used as long as the explanation motion can be determined using skeletal information. For example, it may be determined that an explanation motion is being performed when either P108 or P109 is 0° or more and less than 90°. In addition, it may be determined that an explanation motion is being performed when, for example, a motion of rotating the body or neck by a predetermined amount is performed, a motion of spreading both hands or bringing them to the chest is performed, a motion of pointing up a finger, or the like is performed.
[0025] The region division processing unit A1008 extracts the region of a human body or an object in an image. Specifically, the region division processing unit A1008 performs region division processing using the video information input from the video acquisition unit A1004 to obtain information such as the region and type of the human body or object. The region division processing unit A1008 functions as a first extraction unit that extracts the region of the human body from the image included in the video information (based on the image). The region division processing unit A1008 outputs the acquired information on the region and type of the human body or object to the overlap region extraction unit A1009 as region information. Note that various methods are known for the region division processing performed by the region division processing unit A1008, such as region split, Super-parsing, and full CNN (Convolutional Neural Network) by Deep Learning. In the first embodiment, it is assumed that full CNN is used because it can perform region division with high accuracy, but any method may be used. Since region split, Super-parsing, and fully CNN are well-known technologies, detailed explanations are omitted.
[0026] The overlapping area extraction unit A1009 extracts an overlapping area from the explanatory material. Specifically, the overlapping area is extracted from the explanatory material using the human body movement determination result input from the human body movement determination unit A1007, the skeleton estimation result, the area information input from the area division processing unit A1008, and the explanatory material input from the material acquisition unit A1005. The overlapping area extraction unit A1009 extracts an overlapping area when the determination result input from the human body movement determination unit A1007 indicates that an explanatory movement is being performed. The overlapping area is an area that overlaps with area information of the human body performing the explanatory movement in the area information in the explanatory material. The overlapping area extraction unit A1009 outputs the extracted overlapping area, the explanatory material, and the video information to the transparency change unit A1010. In this embodiment, the overlap area extraction unit A1009 also functions as a second extraction means for extracting an area of explanatory material that overlaps with the human body area as an overlap area based on the human body movement determination result, the human body area, and the explanatory material.
[0027] The overlapping region extraction unit A1009 may combine region information including the region of the human body with the skeleton estimation result to extract a partial region of the human body, such as the face or arms, as an overlapping region. The overlapping region extraction unit A1009 may also extract an overlapping region in the human body even when the determination result indicates that the explanation movement is not being performed.
[0028] The transparency change unit A1010 changes the transparency of at least a part of the explanatory material. Specifically, the transparency of the explanatory material input from the overlap area extraction unit A1009 is changed so as to increase the transparency of the area overlapping the area of the human body, that is, the transparency of the overlap area. The transparency change unit A1010 may change the transparency of the entire explanatory material, or may change the transparency of the so-called blank area in the explanatory material where there are no figures or characters. The transparency degree may also change over time. Any transparency may be used, such as semi-transparent or completely transparent. The transparency change unit A1010 outputs the explanatory material with the changed transparency and the video information to the image superimposition unit A1011. In this embodiment, the transparency change unit A1010 also functions as a transparency change unit that changes the transparency of at least a part of the explanatory material to increase it according to the overlap area.
[0029] The image superimposing unit A1011 superimposes explanatory materials on the video information. Specifically, when explanatory materials with changed transparency and video information are input from the transparency changing unit A1010, the image superimposing unit A1011 superimposes the explanatory materials with changed transparency on the video information. When explanatory materials with changed transparency are not input from the transparency changing unit A1010, the image superimposing unit A1011 superimposes explanatory materials with unchanged transparency (explanatory materials input from the material acquiring unit A1005) on the video information. That is, the image superimposing unit A1011 performs a process of superimposing explanatory materials with different transparency on the image in the video information depending on whether the transparency has been changed or not, and functions as a superimposing unit that generates a superimposed image. The image superimposing unit A1011 outputs the video on which these explanatory materials have been superimposed as a superimposed video to the video output device A1012.
[0030] The video output device A1012 outputs video and image information. Specifically, it outputs the superimposed video input from the image superimposition unit A1011 to the monitor device A1013. In this embodiment, the video output device A1012 also functions as a display control means for displaying the superimposed video composed of the superimposed images on the screen of the monitor device A1013. The monitor device A1013 is a display device that displays the superimposed video input from the video output device A1012 on its screen.
[0031] An example of how the image superimposing unit A1011 superimposes explanatory material on video information will be described below with reference to Figs. 5 to 9. Figs. 5 and 9 are diagrams explaining how an area overlapping with a human body is not made transparent when the human body is not performing an explanatory action. Figs. 5(A) and 9(A) are diagrams showing an example of an image of a human body. Figs. 5(B) and 9(B) are diagrams showing how explanatory material is superimposed on video information. Figs. 5 and 9 show how the image superimposing unit A1011 superimposes explanatory material with its transparency unchanged on video information.
[0032] In FIG. 5, D201 represents an image of a human body. D202 represents an image in which explanatory material is superimposed on D201 (superimposed image). P203 represents explanatory material. P202 represents the human body before it moves, and P202 represents the human body after it moves. In image D201, the human body has simply moved and is not performing an explanatory action, so image superimposition unit A1011 superimposes explanatory material with unchanged transparency on the image of the human body. Therefore, in superimposed image D202, the area of explanatory material P203 that overlaps with the human body area is not made transparent.
[0033] In FIG. 9, D601 represents an image of a human body. D602 represents an image in which explanatory material is superimposed on D601 (superimposed image). P603 represents explanatory material. P601, P602, and P604 each represent a human body. Specifically, P601 represents the human body before it moves, P602 represents the human body after it moves, and P604 represents the human body superimposed on D602. In image D601, the human body has only moved and is not performing an explanatory action, so the image superimposition unit A1011 superimposes explanatory material with its transparency unchanged on the image of the human body. Therefore, in the superimposed image of superimposed image D602, the area of explanatory material P603 that overlaps with the human body area is not made transparent.
[0034] Fig. 6 is a diagram illustrating a state in which an area overlapping a human body is made transparent when the human body is making an explanatory motion. That is, Fig. 6 shows a state in which the image superimposing unit A1011 superimposes explanatory material with a changed transparency on video information. Fig. 6(A) is a diagram showing an example of an image of a human body. Fig. 6(B) is a diagram showing a state in which explanatory material is superimposed on video information.
[0035] In Fig. 6, D301 represents an image of a human body. D302 represents an image (superimposed image) in which explanatory material is superimposed on D301. P301 and P302 represent human bodies. P303 represents explanatory material. Since human body P301 shown in Fig. 6 is performing an explanatory action, image superimposition unit A1011 superimposes explanatory material with a changed transparency on the image of the human body. Therefore, in the superimposed image D302, the area of explanatory material P303 that overlaps with human body P302 (superimposed area) is made transparent, and it is possible to visually confirm which part of the explanatory material human body P301 is explaining.
[0036] Fig. 7 and Fig. 8 are diagrams illustrating a state where an area overlapping a part of a human body is made transparent when the human body is making an explanatory movement. Fig. 7(A) and Fig. 8(A) are diagrams showing examples of images of a human body. Fig. 7(B) and Fig. 8(B) are diagrams showing a state where explanatory material is superimposed on video information. Fig. 7 and Fig. 8 show a state where the image superimposition unit A1011 superimposes explanatory material with a changed transparency on the video information.
[0037] In FIG. 7, D401 represents an image of a human body. D402 represents an image (superimposed image) in which explanatory material is superimposed on D401. P401 represents a human body. P402 represents an arm used in the explanation. P403 represents the explanatory material. Since the human body P401 shown in FIG. 7 is performing an explanation action, the image superimposition unit A1011 superimposes explanatory material with a changed transparency on the image of the human body. Therefore, in the superimposed image D402, the area of the explanatory material P403 that overlaps with the area of the arm P402 of the human body P401 used in the explanation (superimposed area) is made transparent, and it is possible to confirm which part of the explanatory material the human body P401 is explaining.
[0038] In Fig. 8, D501 represents an image of a human body. D502 represents an image (superimposed image) in which explanatory material is superimposed on D501. P501 represents a human body. P502 represents the head of the human body. P503 represents explanatory material. Since the human body P501 shown in Fig. 8 is performing an explanatory action, the image superimposition unit A1011 superimposes explanatory material with a changed transparency on the image of the human body. Therefore, in the superimposed image D502, the area of the explanatory material P503 that overlaps with the area of the head P502 of the human body P501 (superimposed area) is made transparent, making it possible to confirm what the human body P501 is explaining.
[0039] In this way, only when the human body is making an explanatory motion, the overlapping area that overlaps with the human body making the explanatory motion is made transparent, so that when the human body is making an explanatory motion, it is possible to confirm on the screen which part of the explanatory material the human body is explaining. Furthermore, when the human body is not explaining, the entire explanatory material can be confirmed on the screen.
[0040] Fig. 2 is a diagram showing an example of a hardware configuration of the image superimposing device A 1003. As shown in Fig. 2, the image superimposing device A 1003 is composed of a CPU 11, a ROM 12, a RAM 13, a secondary storage device 14, a communication device 15, an input device 16, a communication I / F unit 17, and a connection bus 18.
[0041] The CPU (processor) 11 is a central processing unit that controls the automatic photography system A1000 including the image superimposing device A1003 by executing control programs stored in the ROM 12 and the RAM 13. In other words, it comprehensively controls each component of the automatic photography system A1000 including the image superimposing device A1003. CPU is an acronym for Central Processing Unit.
[0042] The ROM 12 is a non-volatile memory and stores the control program and various parameter data. The control program is executed by the CPU 11 and realizes various processes performed by the image superimposing device A 1003 described later. The RAM 13 is a volatile memory and temporarily stores images, videos, the control program, the execution results thereof, etc.
[0043] The secondary storage device 14 is a rewritable secondary storage device such as a hard disk or flash memory, and stores data received via the communication device 15. It also stores control programs, various settings, processing results, etc. This information is output to the RAM 13 and used by the CPU 11 to execute the programs.
[0044] The communication device 15 is a wired communication unit that communicates with various devices. Note that the communication device 15 is not limited to wired communication, and may be a wireless communication unit. The input device 16 is a mouse, keyboard, touch panel display, etc. that accepts user input.
[0045] The communication I / F unit 17 is an interface for connecting to a removable device, and includes, for example, a power source and an attachment mechanism such as a removable socket for attaching and detaching the removable device. The image superimposing device A1003 performs data communication with the removable device via the communication I / F unit 17. The connection bus 18 connects the CPU 11, ROM 12, RAM 13, secondary storage device 14, communication device 15, input device 16, and communication I / F unit 17 that configure the image superimposing device A1003, and performs data communication between them.
[0046] In this embodiment, the processing in each functional unit is realized by software using the CPU 11 of the image superimposing device A1003, but a part or all of the processing of the image superimposing device A1003 may be realized by hardware. A dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) can be used as the hardware.
[0047] Here, the processing procedure of the automatic photography system A1000 according to the first embodiment will be described with reference to the flowchart in Fig. 10. Fig. 10 is a flowchart showing the processing procedure of the automatic photography system A1000 according to the first embodiment. Each process described below is realized by the CPU 11 of the image superimposition device A1003 executing a program stored in the ROM 12 or the like. Also, by adding an S to the beginning of each process (step), the process (step) is omitted. When activated by a user operation via the input device 16 or the like, the automatic photography system A1000 starts the automatic photography system.
[0048] First, in S001, the video capture unit A1004 captures video information from the video capture device A1001. Then, the process proceeds to S002.
[0049] Next, in S002, the material acquisition unit A1005 acquires explanatory materials from the material acquisition device A1002. After acquiring the explanatory materials, the material acquisition unit A1005 outputs the acquired explanatory materials to the overlap region extraction unit A1009 and the image superimposition unit A1011. Then, the process proceeds to S003.
[0050] Next, in S003, the region division processing unit A1008 performs region division processing using the video information acquired from the video acquisition unit A1004 (first extraction step). Then, the region division processing unit A1008 outputs the divided region information to the overlap region extraction unit A1009. Then, the process proceeds to S004.
[0051] Next, in S004, the skeleton information estimation unit A1006 estimates human skeleton information using the video information acquired from the video acquisition unit A1004. The skeleton information estimation unit A1006 outputs the estimated skeleton information to the human body motion determination unit A1007 as a skeleton estimation result. Then, the process proceeds to S005.
[0052] Next, in S005, the human body movement determination unit A1007 estimates the movement of the human body using the human body skeleton estimation result acquired from the skeleton information estimation unit A1006, and determines whether it is an explanatory movement (first determination step). If the determination result is that it is an explanatory movement (Yes in S005), the human body movement determination unit A1007 outputs the determination result and the skeleton estimation result to the overlapping area extraction unit A1009. Then, the process proceeds to S006. On the other hand, if it is not an explanatory movement (No in S005), the determination result is output to the image superimposition unit A1011. Then, the process proceeds to S008.
[0053] Next, in S006, the overlapping area extraction unit A1009 extracts an area in the explanatory material that overlaps with the human body that has performed the explanatory action (second extraction step). Specifically, based on the determination result and skeleton estimation result input from the human body action determination unit A1007, the human body area information input from the area division processing unit A1008, and the explanatory material input from the material acquisition unit A1005, the overlapping area in the explanatory material that overlaps with the human body that has performed the explanatory action is extracted. The overlapping area extraction unit A1009 outputs the extracted area (overlapping area) and the explanatory material to the transparency change unit A1010. Then, the process proceeds to S007.
[0054] Next, in S007, the transparency change unit A1010 changes the transparency of the explanatory material according to the overlapping area input from the overlapping area extraction unit (changing step). Specifically, the transparency is changed to be higher in the overlapping area. Then, the transparency change unit A1010 outputs the explanatory material with the changed transparency to the image superimposition unit A1011. Then, the process proceeds to S008.
[0055] Next, in S008, the image superimposition unit A1011 superimposes explanatory material on the video information (superimposition step). Here, if a determination result indicating that the human body is not making an explanatory motion is obtained from the human body movement determination unit A1007, the explanatory material obtained from the material acquisition unit A1005 is superimposed on the video information obtained from the human body movement determination unit A1007. On the other hand, if explanatory material with changed transparency is obtained from the transparency change unit A1010, the explanatory material with changed transparency is superimposed on the video information input from the transparency change unit A1010. The image superimposition unit A1011 outputs the superimposed video information (superimposed video) to the video output device A1012. Thereafter, the process proceeds to S009.
[0056] Next, in S009, the video output device A1012 outputs the video information (superimposed video) input from the image superimposition unit A1011 to the monitor device A1013. When the video information is input from the video output device A1012, the monitor device A1013 displays the video or image in the video information on the screen. Then, the process proceeds to S010.
[0057] Next, in S010, it is determined whether or not to end the process. Specifically, it is determined whether or not the user has operated the On / Off switch of the automatic photography system (not shown) to stop the automatic photography process. If the result of the determination is that the automatic photography process has not been stopped (NO in S010), the process proceeds to S001 and the same process is repeated. On the other hand, if the automatic photography process has been stopped (YES in S010), the automatic photography process is ended and this processing flow is ended.
[0058] As described above, the automatic photography system A1000 in the first embodiment can change the transparency of the area of the explanatory material that overlaps with the human body area only when the human body is making an explanatory motion when superimposing explanatory material on the video information. This allows the viewer to see on the screen which part of the explanatory material the human body is explaining when the human body is making an explanatory motion. When the human body is not making an explanatory motion, the entire explanatory material can be seen on the screen. Therefore, the viewer viewing the explanatory material can view the explanatory material more clearly.
[0059] <Embodiment 2> An example of the configuration of an image superimposing device B1003 according to the second embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the functional configuration of an automatic photography system B1000 including an image superimposing device B1003 according to the second embodiment. Note that detailed descriptions of devices and functional unit configurations similar to those of the automatic photography system A1000 in the first embodiment will be omitted below. Also, the hardware configuration is similar to that of the automatic photography system A1000 in the first embodiment, so description thereof will be omitted.
[0060] The automatic photography system B1000 detects a human body from the photographed image, and judges whether the human body is speaking from the audio information acquired from the microphone. If the human body is speaking, it is assumed that the human body is providing an audio explanation, and the system makes transparent (changes the transparency) the area that overlaps with the human body area providing an explanation in the acquired explanatory material, and superimposes it on the video information (photographed human body image). The system then displays the superimposed image on a monitor.
[0061] The automatic photography system B1000 is configured to have an image capturing device A1001, a material capturing device A1002, an image superimposing device B1003, a monitor device A1013, and an audio capturing device B1014. The image superimposing device B1003, the image capturing device A1001, the material capturing device A1002, the monitor device A1013, and the audio capturing device B1014 are communicatively connected. The image superimposing device B1003 and the monitor device A1013 are connected via a line such as a video interface.
[0062] The image superimposing device B1003 acquires human body region information from the video input from the video capturing device A1001, and determines whether or not the human body is speaking from the audio information input from the audio capturing device B1014. If the human body is speaking, it is assumed that the human body is providing an audio explanation, and the image superimposing device B1003 makes the region overlapping with the human body region in the explanatory material transparent and superimposes it on the video information. After the speech ends, the image superimposing device B1003 continues to make the region overlapping with the human body region in the explanatory material transparent for a certain period (predetermined period) assuming that the explanation is continuing. Thereafter, the image superimposing device B1003 outputs the superimposed image, which is the superimposed image, to the monitor device A1013.
[0063] The image superimposing device B1003 is configured to have, as its functional units, a video acquisition unit A1004, a document acquisition unit A1005, a region division processing unit A1008, an overlap region extraction unit A1009, a transparency change unit A1010, and an image superimposing unit B1011. Furthermore, the image superimposing device B1003 is configured to have, as its functional units, a video output device A1012, a sound input unit B1015, an utterance determination unit B1016, an elapsed time measurement unit B1017, and a judgment unit B1018. As in the first embodiment, each of these functional units is realized by the CPU 11 expanding a program stored in the ROM 12 into the RAM 13 and executing it. Then, the CPU 11 stores the execution results of each process described later in the RAM 13 or a predetermined storage medium.
[0064] The voice capturing device B1014 is a device that generates sound information by collecting sounds around the voice capturing device B1014 with a microphone. The voice capturing device B1014 outputs the generated sound information to the sound input unit B1015.
[0065] The sound input unit B1015 receives as input sound information generated by the voice acquisition unit B1014. The sound input unit B1015 outputs the sound information to the speech determination unit B1016 as voice information (voice data).
[0066] The speech determination unit B1016 determines whether or not there is speech from the voice information input from the sound input unit B1015. When making the determination, the speech determination unit B1016 determines that there is a voice explanation if there is speech. To determine whether or not there is speech, voice activity detection (VAD) is used to determine a voice section and other sections from sound data containing voice and other sounds. Since voice section detection is a known technology, detailed description is omitted. The speech determination unit B1016 performs voice section detection on the voice data, and determines that there is speech if a voice section exists. In this embodiment, the speech determination unit B1016 also functions as a second determination means for determining whether or not a human body is speaking based on voice information.
[0067] When it is determined that there is speech, the speech determination unit B1016 outputs first information, which is information that the human body is explaining by voice, to the overlap region extraction unit A1009. On the other hand, when there is no speech, the speech determination unit B1016 outputs second information, which is information that the human body is not speaking, to the elapsed time measurement unit B1017.
[0068] The elapsed time measurement unit B1017 measures the elapsed time after the human body ends its speech. Specifically, the elapsed time measurement unit B1017 measures the elapsed time after the human body ends its speech based on the second information input from the speech determination unit B1016. If the second information is not input, the elapsed time measurement unit B1017 ends the time measurement. The elapsed time measurement unit B1017 outputs the measured elapsed time after the speech ends to the determination unit B1018 as a measured time (elapsed time after the speech ends). In this embodiment, the elapsed time measurement unit B1017 also functions as a measurement means that measures the time after the human body ends its speech based on the second information and outputs the measured time.
[0069] The judgment unit B1018 judges (determines) whether to continue making the explanatory material transparent. Specifically, based on the measured time input from the elapsed time measurement unit B1017, it judges whether to continue making the explanatory material transparent. That is, it judges whether to continue changing the transparency of the overlapping area, which is an area in the explanatory material that overlaps with the human body. If the judgment unit B1018 judges that the explanatory material should be made transparent, it outputs information indicating that the explanatory material should be made transparent to the overlapping area extraction unit A1009. On the other hand, if it judges that the explanatory material should not be made transparent, it outputs information indicating that the explanatory material should not be made transparent to the image superimposition unit B1011. The processing of the judgment unit B1018 in this embodiment will be described below with reference to FIG. 13.
[0070] FIG. 13 is a diagram showing speech intervals of a human body and transparent intervals of a document area according to the second embodiment. In FIG. 13, P801, P802, P803, P804, and P805 respectively represent speech intervals of a human body. P806, P807, P808, P809, and P810 respectively represent the duration of transparentization of explanatory materials for each speech time. In FIG. 13, it is shown that the transparentization starts at the same time as each speech start time and continues for a certain period of time even after each speech ends. In this way, the transparency change unit A1010 in this embodiment continues to change the transparency of the explanatory materials until a predetermined period of time has passed since the speech of the human body ends, based on the measurement time measured by the elapsed time measurement unit B1017. Also, in this embodiment, the certain period (predetermined time) for which the transparentization of the explanatory materials continues after the speech ends is set to 10 seconds. However, this is a guideline, and the transparentization of the explanatory materials may continue for any number of seconds.
[0071] The image superimposing unit B1011 superimposes explanatory materials on the video information. Specifically, when explanatory materials with changed transparency and video information are input from the transparency changing unit A1010, the image superimposing unit B1011 superimposes the explanatory materials with changed transparency on the video information. When explanatory materials with changed transparency are not input from the transparency changing unit A1010, the image superimposing unit B1011 superimposes explanatory materials with unchanged transparency (explanatory materials input from the material acquiring unit A1005) on the video information. When information indicating that the change in transparency is not to be continued is input from the determining unit B1018, the image superimposing unit B1011 superimposes explanatory materials that are not made transparent on the video information. The image superimposing unit B1011 outputs the video information on which these explanatory materials are superimposed to the video output device A1012. Here, an example of the superimposing process of the image superimposing unit B1011 in the second embodiment will be described below with reference to FIG. 12.
[0072] FIG. 12 is a diagram illustrating a state in which explanatory material is superimposed on video information in which an area overlapping with a human body area is made transparent when the human body is speaking according to the second embodiment. In FIG. 12, D701 represents a video in which a human body is photographed. D702 represents a video in which explanatory material is superimposed on the video D701. P701 and P703 each represent a human body. P702 represents an example of a human body speaking. P704 represents explanatory material. Here, the human body P701 shown in FIG. 12 is speaking. Therefore, it is determined that an explanation is being given by speech. Therefore, the image superimposing unit B1011 superimposes explanatory material with a changed transparency on the video information in which the human body P703 is photographed, as shown in explanatory material D702. In this way, in the superimposed video in which explanatory material D702 is superimposed, the area of the explanatory material overlapping with the human body area (superimposed area) is made transparent.
[0073] In this way, when the human body gives a speech explanation, the area of the explanatory material that overlaps with the area of the human body is made transparent, making it possible to visually see which part of the explanatory material the human body is explaining.
[0074] Here, the processing procedure of the automatic photography system B1000 according to the second embodiment will be described with reference to the flowchart in Fig. 14. Fig. 14 is a flowchart showing the processing procedure of the automatic photography system B1000 according to the second embodiment. Each process below is realized by the CPU 11 of the image superimposing device B1003 executing a program stored in the ROM 12 or the like. Also, by adding an S to the beginning of each process (step), the process (step) is omitted. When activated by a user operation via the input device 16 or the like, the automatic photography system B1000 starts the automatic photography system.
[0075] First, in S101, the video acquisition unit A1004 acquires video information from the video acquisition device A1001. Then, the process proceeds to S102.
[0076] Next, in S102, the sound input unit B1015 acquires sound information from the sound acquisition unit B1014. After that, the process proceeds to S103.
[0077] Next, in S103, the material acquisition unit A 1005 acquires explanatory materials from the material acquisition device A 1002. After acquiring the explanatory materials, the material acquisition unit A 1005 outputs the acquired explanatory materials to the overlap region extraction unit A 1009 and the image superimposition unit B 1011. Then, the process proceeds to S104.
[0078] Next, in S104, the region division processing unit A1008 performs region division processing using the video information acquired from the video acquisition unit A1004. Then, the region division processing unit A1008 outputs the divided region information to the overlap region extraction unit A1009. Then, the process proceeds to S104.
[0079] Next, in S105, the speech determination unit B1016 performs speech section detection using the voice information input from the sound input unit B1015, and determines whether or not the human body is explaining by voice. If the result of the determination is that the human body is explaining by voice (Yes in S105), the speech determination unit B1016 outputs information (first information) that the human body is explaining by voice to the overlap region extraction unit A1009. Then, the process proceeds to S108. On the other hand, if there is no explanation by voice (No in S105), the speech determination unit B1016 outputs information (second information) that the human body is not speaking to the elapsed time measurement unit B1017. Then, the process proceeds to S106.
[0080] Next, in S106, the elapsed time measurement unit B1017 measures the elapsed time from the end of the utterance based on the second information input from the utterance determination unit B1016. Then, the elapsed time measurement unit B1017 outputs the measured time to the determination unit B1018. Then, the process proceeds to S107.
[0081] Next, in S107, the judgment unit B1018 judges whether the elapsed time since the end of the utterance input from the elapsed time measurement unit B1017 has exceeded a certain period of time. If the judgment result shows that the elapsed time has exceeded a certain period of time (Yes in S107), information not to change the transparency (not to continue the transparency) is output to the image superimposition unit B1011. Then, the process proceeds to S110. On the other hand, if the elapsed time has not exceeded a certain period of time (No in S107), information to change the transparency (to continue the transparency) is output to the overlapped area extraction unit A1009. Then, the process proceeds to S108.
[0082] Next, in S108, the overlap region extraction unit A1009 extracts an overlap region from the explanatory material. Specifically, when the first information is input from the speech determination unit B1016 or the information for changing the transparency is input from the determination unit B1018, the overlap region is extracted using the human body region information input from the region division processing unit and the explanatory material input from the material acquisition unit A1005. Then, the overlap region extraction unit A1009 outputs the extracted overlap region to the transparency change unit A1010. Then, the process proceeds to S109.
[0083] Next, in S109, the transparency changing unit A1010 changes the transparency of the explanatory material (makes it transparent) using the explanatory material and the overlapping area input from the overlapping area extraction unit A1009. Then, the transparency changing unit A1010 outputs the explanatory material with the changed transparency to the image superimposition unit B1011. Then, the process proceeds to S110.
[0084] Next, in S110, the image superimposing unit B1011 superimposes explanatory material on the video information. Here, when explanatory material with changed transparency is input from the transparency changing unit A1010, the image superimposing unit B1011 superimposes the explanatory material with changed transparency on the video information. When explanatory material with changed transparency is not input from the transparency changing unit A1010, the image superimposing unit B1011 superimposes explanatory material with unchanged transparency acquired from the material acquiring unit A1005 on the video information. When information on unchanged transparency is input from the determining unit B1018, the image superimposing unit B1011 superimposes explanatory material with unchanged transparency acquired from the material acquiring unit A1005 on the video information. Then, the image superimposing unit B1011 outputs the superimposed image to the video output device A1012. Then, the process proceeds to S111.
[0085] Next, in S111, the video output device A 1012 outputs the video information (superimposed video) input from the image superimposition unit B 1011 to the monitor device A 1013. When the video information is input from the video output device A 1012, the monitor device A 1013 displays the video or image in the video information on the screen. Then, the process proceeds to S112.
[0086] Next, in S112, it is determined whether or not to end the process. Specifically, it is determined whether or not the user has operated the On / Off switch of the automatic photography system (not shown) to stop the automatic photography process. If the result of the determination is that the automatic photography process has not been stopped (NO in S112), the process proceeds to S101 and the same process is repeated. On the other hand, if the automatic photography process has been stopped (YES in S112), the automatic photography process is ended and this processing flow is ended.
[0087] As described above, the automatic photography system B1000 in the second embodiment can change the transparency of the area of the explanatory material that overlaps with the human body area only when the human body is giving an explanation by voice, when the automatic photography system B1000 in the second embodiment superimposes explanatory material on the video information. This makes it possible to visually confirm on the screen which part of the explanatory material the human body is explaining when the human body is giving an explanation. And when the human body is not giving an explanation, the entire explanatory material can be visually confirmed on the screen.
[0088] <Embodiment 3> An example of the configuration of an image superimposing device C1000 according to the third embodiment will be described with reference to Fig. 15. Fig. 15 is a block diagram showing the functional configuration of an automatic photography system C1000 including an image superimposing device C1000 according to the third embodiment. Note that detailed descriptions of the configuration of devices and functional units similar to those of the automatic photography systems A1000 and B1000 in the first and second embodiments will be omitted below. Furthermore, the hardware configuration is also similar to that of the automatic photography system A1000 in the first embodiment, and therefore description thereof will be omitted.
[0089] The automatic photography system C1000 detects a human body from the captured image, and determines whether the human body is speaking from the audio information acquired from the microphone. If the human body is speaking, it extracts a predetermined keyword from the audio information and explanatory materials. If the keywords from both are the same or similar, it determines that the human body is providing an audio explanation, and makes transparent (changes the transparency) the area that overlaps with the human body area providing an explanation in the acquired explanatory materials, and superimposes it on the image information. The processing system then displays the superimposed image on a monitor.
[0090] The automatic photography system C1000 is configured to have an image capturing device A1001, a material capturing device A1002, an image superimposing device C1003, a monitor device A1013, and an audio capturing device B1014. The image superimposing device C1003 is communicably connected to the image capturing device A1001, the material capturing device A1002, the monitor device A1013, and the audio capturing device B1014. The image superimposing device C1003 and the monitor device A1013 are connected via a line such as a video interface.
[0091] The image superimposing device C1003 acquires region information of the human body from the video input from the video capturing device A1001, and judges whether the human body is speaking from the voice information input from the voice capturing device B1014. It then extracts a predetermined keyword from the voice information and explanatory material, and if the keyword matches or is similar, it assumes that the human body is providing a voice explanation, changes the transparency of the region in the explanatory material that overlaps with the human body region, and superimposes it on the video information. It then continues to change the transparency of the region in the explanatory material that overlaps with the human body region (overlapping region) for a certain period after the end of the speech, assuming that the explanation is continuing. The image superimposing device C1003 then outputs the superimposed image, which is the superimposed image, to the monitor device A1013.
[0092] The image superimposing device C1003 is configured to have, as its functional units, a video acquisition unit A1004, a document acquisition unit A1005, a region division processing unit A1008, an overlap region extraction unit A1009, a transparency change unit A1010, an image superimposing unit B1011, and a video output device A1012. Furthermore, the image superimposing device B1003 is configured to have, as its functional units, a sound input unit B1015, an utterance determination unit B1016, an elapsed time measurement unit C1017, a judgment unit B1018, a keyword extraction unit C1019, and a match determination unit C1020. Each of these functional units is realized by the CPU 11 expanding a program stored in the ROM 12 into the RAM 13 and executing it, as in the first embodiment. Then, the CPU 11 stores the execution results of each process, which will be described later, in the RAM 13 or a predetermined storage medium.
[0093] The keyword extraction unit C1019 extracts keywords from both the voice information and the explanatory materials. Specifically, it extracts predetermined keywords from the voice information input from the speech determination unit B1016 and the explanatory materials input from the material acquisition unit A1005. From the voice information, it extracts keywords by voice recognition technology. From the explanatory materials, it extracts keywords by OCR (Optical Character Recognition) or from tag information embedded in advance in the explanatory materials. The keyword extraction unit C1019 outputs the extracted keywords to the match determination unit C1020. In this embodiment, the keyword extraction unit C1019 functions as a third extraction means for extracting keywords from both the voice information and the explanatory materials.
[0094] The match determination unit C1020 determines whether the keywords extracted by the keyword extraction unit C1019 match or are similar. Specifically, it determines whether the keywords extracted from the speech content input from the keyword extraction unit C1019 match or are similar to the keywords extracted from the explanatory material. If the speech content and the explanatory content match or are similar, the match determination unit C1020 outputs third information, which is information on the match or similarity, to the overlap area extraction unit A1009. On the other hand, if the speech content and the explanatory content do not match or are not similar, the match determination unit C1020 outputs fourth information, which is information on the match or dissimilarity, to the elapsed time measurement unit C1017 and the image superimposition unit B1011. In this embodiment, the match determination unit C1020 also functions as a third determination means for determining whether the keywords extracted by the keyword extraction unit C1019 match or are similar.
[0095] Fig. 16 is an explanatory diagram for identifying an explanatory area from the contents of the speech of a human body and the contents of explanatory materials according to the third embodiment. Fig. 16(A) is a diagram showing an example of explanatory materials and areas in the explanatory materials. Fig. 16(B) is a diagram showing an example of keywords extracted from speech. The explanatory area is an area showing explanatory figures, tables, images, videos, animations, characters, etc. displayed in the explanatory materials.
[0096] In FIG. 16(A), P901 represents explanatory material. P902 is an area in the explanatory material, and represents area 1 in the table. P903 is an area in the explanatory material, and represents area 2 in the table. In FIG. 16(B), the table shows keywords extracted from the utterance, specifically, keywords extracted from area P902, which is area 1, and area P903, which is area 2. Using FIG. 16 as an example, when comparing the keywords of the utterance content with the keywords of area P902, which is area 1, and area P903, which is area 2, the keywords of area and area match the keywords of area P903, which is area 2. Therefore, in the example shown in FIG. 16, the match determination unit C1020 can determine that a human body is explaining area P903, which is area 2, by voice.
[0097] The elapsed time measurement unit C1017 measures the elapsed time since the human body's speech ended. Specifically, based on the second information input from the speech determination unit B1016 and the fourth information input from the match determination unit C1020, it measures the elapsed time since the speech that matches the contents of the explanatory material ended. If the second information is not input, the time measurement ends. Then, the elapsed time measurement unit C1017 outputs the measured time to the determination unit B1018.
[0098] Here, the processing procedure of the automatic photography system C1000 according to the third embodiment will be described with reference to the flowcharts of Fig. 17 and Fig. 18. Fig. 17 and Fig. 18 are flowcharts showing the processing procedure of the automatic photography system C1000 according to the third embodiment. Each process below is realized by the CPU 11 of the image superimposition device C1003 executing a program stored in the ROM 12 or the like. Also, by adding an S to the beginning of each process (step), the process (step) is omitted. When activated by a user operation via the input device 16 or the like, the automatic photography system C1000 starts the automatic photography system.
[0099] First, in S201, the video acquisition unit A1004 acquires video information from the video acquisition device A1001. Then, the process proceeds to S202.
[0100] Next, in S202, the sound input unit B1015 acquires sound information from the sound acquisition unit B1014. After that, the process proceeds to S203.
[0101] Next, in S203, the material acquisition unit A 1005 acquires explanatory materials from the material acquisition device A 1002. After acquiring the explanatory materials, the material acquisition unit A 1005 outputs the acquired explanatory materials to the overlap region extraction unit A 1009 and the image superimposition unit B 1011. Then, the process proceeds to S204.
[0102] Next, in S204, the region division processing unit A1008 performs region division processing using the video information acquired from the video acquisition unit A1004. Then, the region division processing unit A1008 outputs the divided region information to the overlap region extraction unit A1009. Then, the process proceeds to S204.
[0103] Next, in S205, the speech determination unit B1016 performs speech section detection using the voice information input from the sound input unit B1015, and determines whether or not the human body is providing a voice explanation. If the result of the determination is that the human body is providing a voice explanation (Yes in S205), the speech determination unit B1016 outputs information (first information) that the human body is providing a voice explanation to the keyword extraction unit C1019. Then, the process proceeds to S206. On the other hand, if the human body is not providing a voice explanation (No in S205), the process outputs information (second information) that the human body is not providing a voice explanation to the elapsed time measurement unit C1017. Then, the process proceeds to S209.
[0104] Next, in S206, the keyword extraction unit C1019 extracts a keyword from the voice information input from the speech determination unit B 1016. Then, the keyword extraction unit C1019 outputs the extracted keyword to the match determination unit C1020. After that, the process proceeds to S207.
[0105] Next, in S207, the keyword extraction unit C1019 extracts keywords from the explanatory materials input from the material acquisition unit A1005. The keyword extraction unit C1019 outputs the extracted keywords to the match determination unit C1020. Then, the process proceeds to S208. Note that the order of the processes of S206 and S207 may be reversed.
[0106] Next, in S208, the match determination unit C1020 determines whether the voice information input from the keyword extraction unit C1019 and the keyword extracted from the explanatory material match or are similar. If the result of the determination is that the voice information and the keyword match or are similar (Yes in S208), the match determination unit C1020 outputs information that matches the explanatory content (third information) to the overlap area extraction unit A1009. Then, the process proceeds to S211. On the other hand, if the voice information and the keyword do not match or are not similar (No in S208), the match determination unit C1020 outputs information that does not match the explanatory content (fourth information) to the image superimposition unit B1011 and the elapsed time measurement unit C1017. Then, the process proceeds to S213.
[0107] Next, in S209, the elapsed time measurement unit C1017 measures the elapsed time from the end of the utterance whose content does not match the explanatory material, based on the second information input from the utterance determination unit B1016 and the fourth information input from the match determination unit C1020. The elapsed time measurement unit C1017 outputs the measured time (measured time) to the determination unit B1018. Then, the process proceeds to S210.
[0108] Next, in S210, the determination unit B1018 determines whether the utterance that does not match the explanatory material has ended or whether a certain period of time has passed since the utterance that matches the explanatory material ended, based on the measured time input from the elapsed time measurement unit C1017. If the result of the determination is that the utterance that does not match the explanatory material has ended or the certain period of time has passed (Yes in S210), the determination unit B1018 outputs information that does not make the explanatory material transparent to the image superimposition unit B1011. Then, the process proceeds to S213. On the other hand, if the utterance that does not match the explanatory material has not ended or the certain period of time has not passed (No in S210), the determination unit B1018 outputs information that makes the explanatory material transparent to the overlapping area extraction unit A1009. Then, the process proceeds to S211.
[0109] Next, in S211, the overlap region extraction unit A1009 extracts an overlap region using the human body region information input from the region division processing unit A1008 and the explanatory materials input from the material acquisition unit A1005. Then, the overlap region extraction unit A1009 outputs the extracted overlap region to the transparency change unit A1010. After that, the process proceeds to S212.
[0110] Next, in S212, the transparency changing unit A1010 changes the transparency of the explanatory material (makes it transparent) using the explanatory material and the overlapping area input from the overlapping area extraction unit A1009. Then, the transparency changing unit A1010 outputs the explanatory material with the changed transparency to the image superimposition unit B1011. Then, the process proceeds to S213.
[0111] Next, in S213, the image superimposing unit B1011 superimposes the explanatory material on the video information. Here, when the explanatory material with the changed transparency is input from the transparency changing unit A1010, the explanatory material with the changed transparency is superimposed on the video information. Also, when the information with the unchanged transparency is input from the determining unit B1018, the explanatory material with the unchanged transparency input from the material acquiring unit A1005 is superimposed on the video information. Also, when the information (fourth information) with which the spoken content and the explanatory material do not match is input from the matching determining unit C1020, the explanatory material with the unchanged transparency input from the material acquiring unit A1005 is superimposed on the video information. Then, the image superimposing unit B1011 outputs the superimposed video to the video output device A1012. Thereafter, the process proceeds to S214.
[0112] Next, in S214, the video output device A 1012 outputs the video information (superimposed video) input from the image superimposition unit B 1011 to the monitor device A 1013. When the video information is input from the video output device A 1012, the monitor device A 1013 displays the video or image in the video information on the screen. Then, the process proceeds to S215.
[0113] Next, in S215, it is determined whether or not to end the process. Specifically, it is determined whether or not the user has operated the On / Off switch of the automatic photography system (not shown) to stop the automatic photography process. If the result of the determination is that the automatic photography process has not been stopped (NO in S215), the process proceeds to S201 and the same process is repeated. On the other hand, if the automatic photography process has been stopped (YES in S215), the automatic photography process is ended and this processing flow is ended.
[0114] As described above, when the automatic photography system A1000 in the third embodiment superimposes explanatory material on video information, the transparency of the area of the explanatory material that overlaps with the human body area is changed only when the explanation by the voice of the human body matches or is similar to the content of the explanatory material. This allows the user to visually confirm on the screen which part of the explanatory material the human body is explaining when the human body is explaining, and allows the user to visually confirm the entire explanatory material when the human body is not explaining.
[0115] <Embodiment 4> An example of the configuration of an image superimposing device D1003 according to the fourth embodiment will be described with reference to Fig. 19. Fig. 19 is a block diagram showing the functional configuration of an automatic photography system B1000 including an image superimposing device D1003 according to the fourth embodiment. Note that detailed descriptions of the configuration of devices and functional units similar to those of the automatic photography systems A1000, B1000, and C1000 in the first, second, and third embodiments will be omitted below. Also, the hardware configuration is similar to that of the automatic photography system A1000 in the first embodiment, so description thereof will be omitted.
[0116] The automatic photography system D1000 detects a human body from the captured image. Then, when it determines that the human body is making an explanatory movement based on the skeletal information of the human body, and when it determines that an explanation is being given by voice based on the similarity between the audio information acquired from the microphone and the contents of the explanatory material, if the area of the human body overlaps with the explanatory material, it performs an emphasis process on the explanatory material. On the other hand, if the area of the human body does not overlap with the explanatory material, the processing system makes the overlapping area, which is the area of the explanatory material that overlaps with the human body, transparent (changes the transparency), and superimposes it on the captured image of the human body, and displays the result on a monitor.
[0117] The automatic photography system D1000 is configured to have an image capturing device A1001, a material capturing device A1002, an image superimposing device D1003, a monitor device A1013, and an audio capturing device B1014. The image superimposing device D1003 is communicatively connected to the image capturing device A1001, the material capturing device A1002, the monitor device A1013, and the audio capturing device B1014. The image superimposing device D1003 and the monitor device A1013 are connected via a line such as a video interface.
[0118] The image superimposing device D1003 detects a human body from the video input from the video capture device A1001, and judges whether an explanatory action is being performed from skeletal information of the detected human body. Then, when audio information is input from the audio capture device B1014, it judges whether a human body is speaking from the audio and judges whether it matches the contents of the explanatory material. If it is judged that an explanation is being performed by action or audio, it identifies the explanatory area being explained from the explanatory material and judges whether the explanatory area overlaps with the human body area. If the human body area and the explanatory area overlap, it performs a highlighting process on the explanatory area without performing a transparency process on the overlapping area and the explanatory area, which are the areas of the explanatory material that overlap with the human body. If they do not overlap, it makes the overlapping area transparent (changes the transparency) and superimposes it on the video information. It outputs the superimposed image to the monitor device A1013.
[0119] The image superimposing device D1003 is configured to have, as its functional units, a video acquisition unit A1004, a document acquisition unit A1005, a region division processing unit A1008, an overlap region extraction unit A1009, a transparency change unit A1010, a video output device A1012, and a sound input unit B1015. Furthermore, the image superimposing device D1003 is configured to have, as its functional units, an utterance determination unit B1016, an elapsed time measurement unit C1017, a transparency continuation determination unit C1018, a keyword extraction unit C1019, and a match determination unit C1020. Furthermore, the image superimposing device D1003 is configured to have, as its functional units, a description region identification unit D1021, an overlap determination unit D1022, a highlight frame superimposing unit D1023, and an image superimposing unit D1011. Each of these functional units is realized by the CPU 11 expanding a program stored in the ROM 12 into the RAM 13 and executing it, as in the first embodiment. Then, the CPU 11 stores the results of each process, which will be described later, in the RAM 13 or a predetermined storage medium.
[0120] The explanatory region identification unit D1021 identifies an explanatory region that is a region that the human body is targeting for the explanatory action from the explanatory material. Specifically, the explanatory region identification unit D1021 identifies the region that the human body is targeting for the explanatory action using human body skeletal information, video information, region information, the explanatory material, and information on the explanatory region that matches the audio explanation. The skeletal information and video information are input from the human body action determination unit A1007. Also, the region information is input from the region division processing unit A1008. The explanatory material is input from the material acquisition unit A1005. Information on the explanatory region that matches the audio explanation is input from the match determination unit C1020. The explanatory region identification unit D1021 outputs the identified explanatory region, region information, video information, and explanatory material to the overlap determination unit D1022.
[0121] FIG. 20 is a diagram showing how to specify an explanation region from the explanation action of a human body according to the fourth embodiment. In FIG. 20, P1001 represents a human body. P1002 and P1003 represent regions within the explanation region. P1004 represents a half line passing through the arm of the human body that is giving the explanation. When the half line P1004 and each region have an intersection, the region is specified as the region to be explained (explanation region). Whether or not a line and a rectangular region have an intersection can be determined by an intersection determination. In FIG. 20, the half line P1004 and region P1003 have an intersection, so that the region where the human body is giving the explanation can be specified. When information on the explanation region that matches the explanation by voice is input, it is determined as the region to be explained (explanation region) that has been specified.
[0122] The overlap determination unit D1022 determines whether the region of the human body overlaps with the identified explanatory region. Specifically, it determines whether the region of the human body overlaps with the identified explanatory region using the identified explanatory region, region information, video information, and explanatory materials input from the explanatory region determination unit D1021. If the result of the determination is that the region of the human body overlaps with the identified explanatory region, the overlap determination unit D1022 outputs the identified explanatory region, video information, and explanatory materials to the highlight frame superimposition unit D1023. On the other hand, if the region of the human body does not overlap with the identified explanatory region, it outputs the region information, video information, and explanatory materials to the overlap region extraction unit A1009.
[0123] The highlight frame superimposing unit D1023 highlights the explanatory region of the explanatory material. Specifically, the highlight frame is superimposed on the explanatory region of the explanatory material using the identified explanatory region, video information, and explanatory material input from the overlap determination unit D1022. The highlight frame superimposing unit D1023 outputs the superimposed explanatory material and video information to the image superimposing unit D1011. The highlight frame to be highlighted may be colored, and the thickness of the frame may be changeable. The color within the explanatory region may be changed to a color different from that outside the explanatory region, and the color, font, and size of the figures and characters in the explanatory region may be changed. The highlight frame may be made to blink, or a combination of these may be used for highlighting.
[0124] The image superimposing unit D1011 superimposes explanatory material on the video information. Specifically, when explanatory material with an emphasis frame superimposed thereon input from the emphasis frame superimposing unit D1023 and explanatory material with an area overlapping with the human body input from the video information and the transparency changing unit A1010 made transparent are input, the image superimposing unit D1011 superimposes explanatory material with the transparency changed on the video information. When explanatory material with the transparency changed is not input from the transparency changing unit A1010, explanatory material with the transparency not changed (explanatory material input from the material acquiring unit A1005) is superimposed on the video information. When information indicating that the change in transparency is not to be continued is input from the determining unit B1018, explanatory material that is not made transparent is superimposed on the video information. The image superimposing unit D1011 outputs the video information with these explanatory materials superimposed thereon to the video output device A1012. Here, an example of the superimposition process of the image superimposition unit D1011 of the third embodiment will be described below with reference to FIGS.
[0125] Fig. 21 is a diagram showing an example of a case where the human body and the explanatory region do not overlap. Fig. 21(A) is a diagram showing an example of video information. Fig. 21(B) is a diagram showing an example of explanatory material. Fig. 21(C) is a diagram showing an example of overlap between the human body region and the explanatory material region. Fig. 21(D) is a diagram showing an example of the region to be explained (explanation region).
[0126] In FIG. 21, D1101 represents video information. D1102 represents explanatory material. D1103 is a diagram showing the overlap between the human body region and the explanatory material region. D1104 represents video information in which explanatory material D1102 is superimposed on video information D1101. P1101 represents a human body. P1102 represents a human body region. P1103 represents a region that is not the subject of explanation. P1104 represents an explanatory region. P1105, P1106, and P1107 are similar to P1102, P1103, and P1104 respectively, so their explanations will be omitted. P1108 represents the same human body as P1101. P1109 represents explanatory material. In this case, the human body region P1105 and the explanatory region P1107 do not overlap.
[0127] Therefore, in D1104, the area (overlapping area) that overlaps with the human body area in explanatory material P1109 is made transparent and is overlaid on the video information. As such, in Fig. 21, since the human body and the explanation area do not overlap, the explanation area can be seen even if the explanation area that overlaps with the human body area is made transparent.
[0128] Fig. 22 is a diagram showing an example of a case where a human body and an explanatory region overlap. Fig. 22(A) is a diagram showing an example of video information. Fig. 22(B) is a diagram showing an example of explanatory material. Fig. 22(C) is a diagram showing an example of overlap between a human body region and an explanatory material region. Fig. 22(D) is a diagram showing an example of the region to be explained (explanation region).
[0129] In FIG. 22, D1201 represents video information. D1202 represents explanatory material. D1203 is a diagram showing the overlap between the human body region and the explanatory material region. D1204 represents video information in which explanatory material D1202 is superimposed on video information D1201. P1201 represents a human body. P1202 represents a human body region. P1203 represents a region that is not the subject of explanation. P1204 represents an explanatory region. P1205, P1206, and P1207 are similar to P1202, P1203, and P1204, respectively, and therefore their explanations will be omitted. P1208 represents a highlighted frame superimposed on the explanatory region. P1209 represents explanatory material. At this time, the human body region P1205 and the explanatory region P1207 overlap.
[0130] Therefore, in D1204, a highlight frame P1208 is superimposed on the explanation region in the explanation region P1209. In this way, when the human body that has performed the explanation through action (explanatory action) overlaps with the explanation region, the explanation region can be confirmed by performing highlighting processing on the explanation region even if the human body cannot be visually recognized.
[0131] Fig. 23 is a diagram showing an example of a case where a human body and an explanatory region overlap. Fig. 23(A) is a diagram showing an example of video information. Fig. 23(B) is a diagram showing an example of explanatory material. Fig. 23(C) is a diagram showing an example of overlap between a human body region and an explanatory material region. Fig. 23(D) is a diagram showing an example of the region to be explained (explanation region).
[0132] In FIG. 23, D1301 represents video information. D1302 represents explanatory material. D1303 is a diagram showing the overlap between the human body region and the explanatory material region. D1304 represents video information in which explanatory material D1302 is superimposed on video information D1301. P1301 represents a human body. P1302 represents the human body region, and P1303 represent regions that are not the subject of explanation. P1304 represents the explanatory region. P1305, P1306, and P1307 are similar to P1302, P1303, and P1304, respectively, and therefore their explanation will be omitted. P1308 represents a highlighted frame superimposed on the explanatory region. P1309 represents explanatory material. In this case, the human body region P1305 and the explanatory region P1307 overlap.
[0133] Therefore, in D1304, a highlight frame is superimposed on the explanation region in the explanatory material P1309. When the human body providing the audio explanation overlaps with the explanation region in this way, highlighting the explanation region allows the explanation region to be confirmed even if the human body cannot be seen.
[0134] Here, the procedure for processing the automatic photography system D1000 will be described with reference to the flowcharts of Fig. 24 and Fig. 25. Fig. 24 and Fig. 25 are flowcharts showing the processing procedure of the automatic photography system D1000 according to the fourth embodiment. Each process below is realized by the CPU 11 of the image superimposing device D1003 executing a program stored in the ROM 12 or the like. Also, by adding an S to the beginning of each process (step), the process (step) is omitted. When activated by a user operation via the input device 16 or the like, the automatic photography system D1000 starts the automatic photography system.
[0135] First, in S301, the video acquisition unit A1004 acquires video information from the video acquisition device A1001. Then, the process proceeds to S302.
[0136] Next, in S302, the sound input unit B1015 acquires sound information from the sound acquisition unit B1014. After that, the process proceeds to S303.
[0137] Next, in S303, the material acquisition unit A 1005 acquires explanatory materials from the material acquisition device A 1002. After acquiring the explanatory materials, the material acquisition unit A 1005 outputs the acquired explanatory materials to the overlap region extraction unit A 1009 and the image superimposition unit B 1011. Then, the process proceeds to S304.
[0138] Next, in S304, the region division processing unit A1008 performs region division processing using the video information acquired from the video acquisition unit A1004. Then, the region division processing unit A1008 outputs the divided region information to the overlap region extraction unit A1009. After that, the process proceeds to S305.
[0139] Next, in S305, the skeleton information estimation unit A1006 estimates human skeleton information using the video information acquired from the video acquisition unit A1004. The skeleton information estimation unit A1006 outputs the estimated skeleton information as a skeleton estimation result to the human body motion determination unit A1007. Then, the process proceeds to S306.
[0140] Next, in S306, the human body motion determination unit A1007 estimates a human body motion using the human body skeleton estimation result acquired from the skeleton information estimation unit A1006, and determines whether or not the motion is an explanatory motion. If the determination result indicates that the motion is an explanatory motion (Yes in S306), the human body motion determination unit A1007 outputs the determination result and the skeleton estimation result to the explanatory region identification unit D1021. Then, the process proceeds to S313. On the other hand, if the motion is not an explanatory motion (No in S005), the determination result is output to the image superimposition unit D1011. Then, the process proceeds to S307.
[0141] Next, in S307, the speech determination unit B1016 performs speech section detection using the voice information input from the sound input unit B1015, and determines whether or not the human body is providing a voice explanation. If the result of the determination is that the human body is providing a voice explanation (Yes in S307), the speech determination unit B1016 outputs information (first information) that the human body is providing a voice explanation to the keyword extraction unit C1019. Then, the process proceeds to S308. On the other hand, if the human body is not providing a voice explanation (No in S307), the process outputs information (second information) that the human body is not providing a voice explanation to the elapsed time measurement unit C1017. Then, the process proceeds to S311.
[0142] Next, in S308, the keyword extraction unit C1019 extracts keywords from the voice information input from the utterance determination unit B 1016. Then, the keyword extraction unit C1019 outputs the extracted keywords to the match determination unit C1020. After that, the process proceeds to S309.
[0143] Next, in S309, the keyword extraction unit C1019 extracts keywords from the explanatory materials input from the material acquisition unit A1005. The keyword extraction unit C1019 outputs the extracted keywords to the match determination unit C1020. After that, the process proceeds to S310. Note that the order of the processes of S308 and S309 may be reversed.
[0144] Next, in S310, the match determination unit C1020 determines whether the voice information input from the keyword extraction unit C1019 and the keyword extracted from the explanatory material match or are similar. If the result of the determination is that the voice information and the keyword match or are similar (Yes in S310), the match determination unit C1020 outputs information (third information) that matches the explanatory content to the explanatory region identification unit D1021. Then, proceed to S313. On the other hand, if the voice information and the keyword do not match or are not similar (No in S310), the match determination unit C1020 outputs information (fourth information) that does not match the explanatory content to the image superimposition unit D1011. Then, proceed to S318.
[0145] Next, in S311, the elapsed time measurement unit C1017 measures the elapsed time from the end of the utterance whose content does not match the explanatory material, based on the second information input from the utterance determination unit B1016 and the fourth information input from the match determination unit C1020. The elapsed time measurement unit C1017 outputs the measured time (measured time) to the determination unit B1018. Then, the process proceeds to S312.
[0146] Next, in S312, the determination unit B1018 determines whether or not a certain period of time has elapsed since the speech that does not match the explanatory material ended or the speech that matches the explanatory material ended, based on the measured time input from the elapsed time measurement unit C1017. If the result of the determination is that the speech that does not match the explanatory material ended or the certain period of time has elapsed (Yes in S312), the determination unit B1018 outputs information that does not make the explanatory material transparent to the image superimposition unit D1011. Then, the process proceeds to S318. On the other hand, if the speech that does not match the explanatory material has not ended or the certain period of time has not elapsed (No in S312), the determination unit B1018 outputs information that makes the explanatory material transparent to the overlapping region extraction unit A1009. Then, the process proceeds to S313.
[0147] Next, in S313, the explanatory region identification unit D1021 identifies the region where the human body is explaining (explanatory region). When identifying the region where the human body is explaining, the identification is performed using information on the human body skeleton estimation result, video information, explanatory materials, and information that matches the explanatory content (third information). The human body skeleton estimation result and video information are input from the human body motion determination unit A1007. The region information is input from the region division processing unit A1008. The explanatory materials are input from the material acquisition unit A1005. The information that matches the explanatory content (third information) is input from the match determination unit C1020. Then, the explanatory region identification unit D1021 outputs the region information, the identified explanatory region, the explanatory materials, and the video information to the overlap determination unit D1022. Thereafter, the process proceeds to S314.
[0148] Next, in S314, the overlap determination unit D1022 determines whether the human body region and the explanatory region overlap from the region information input from the explanatory region identification unit D1021, the identified explanatory region, explanatory materials, and video information. If the human body region and the explanatory region overlap (Yes in S314), the overlap determination unit D1022 outputs the identified explanatory region, video information, and explanatory materials to the highlight frame superimposition unit D1023. Then, proceed to S315. On the other hand, if the human body region and the explanatory region do not overlap (No in S314), the overlap determination unit D1022 outputs the video information, explanatory materials, and region information to the overlap region extraction unit A1009. Then, proceed to S316.
[0149] Next, in S315, the highlight frame superimposition unit D1023 superimposes a highlight frame on the specified explanatory area of the explanatory material using the specified explanatory area, video information, and explanatory material input from the overlap determination unit D1022. The highlight frame superimposition unit D1023 outputs the superimposed explanatory material and video information to the image superimposition unit D1011. Then, the process proceeds to S318.
[0150] Next, in S316, the overlap region extraction unit A1009 extracts an overlap region using the human body region information input from the overlap determination unit D1022 and the explanatory materials input from the material acquisition unit A1005. Then, the overlap region extraction unit A1009 outputs the extracted overlap region to the transparency change unit A1010. Then, the process proceeds to S317.
[0151] Next, in S317, the transparency changing unit A1010 changes the transparency of the explanatory material (makes it transparent) using the explanatory material and the overlapping area input from the overlapping area extraction unit A1009. Then, the transparency changing unit A1010 outputs the explanatory material with the changed transparency to the image superimposition unit B1011. Then, the process proceeds to S318.
[0152] Next, in S318, the image superimposing unit D1011 superimposes explanatory material on the video information. Here, when explanatory material with changed transparency is input from the transparency changing unit A1010, the image superimposing unit D1011 superimposes the explanatory material with changed transparency on the video information. When information with no change in transparency is input from the determining unit B1018, the image superimposing unit D1011 superimposes explanatory material with unchanged transparency input from the material acquiring unit A1005 on the video information. When information (fourth information) with which the spoken content and the content of the explanatory material do not match is input from the matching determining unit C1020, the image superimposing unit D1011 superimposes explanatory material with unchanged transparency input from the material acquiring unit A1005 on the video information. When an explanatory region with a highlighted frame superimposed is input from the highlight frame superimposing unit D1023, the image superimposing unit D1011 superimposes explanatory material with a highlighted frame superimposed on the video information. Then, the image superimposing unit D1011 outputs the superimposed video to the video output device A1012. Then proceed to S319.
[0153] Next, in S319, the video output device A 1012 outputs the video information (superimposed video) input from the image superimposition unit B 1011 to the monitor device A 1013. When the video information is input from the video output device A 1012, the monitor device A 1013 displays the video or image in the video information on the screen. Then, the process proceeds to S320.
[0154] Next, in S320, it is determined whether or not to end the process. Specifically, it is determined whether or not the user has operated the On / Off switch of the automatic photography system (not shown) to stop the automatic photography process. If the result of the determination is that the automatic photography process has not been stopped (NO in S320), the process proceeds to S301 and the same process is repeated. On the other hand, if the automatic photography process has been stopped (YES in S320), the automatic photography process is ended and this processing flow is ended.
[0155] As described above, the automatic photography system A1000 in the fourth embodiment superimposes explanatory material on a video of a human body, and if the human body and the area of the explanation subject overlap when the human body is explaining by voice or movement, a highlight frame is superimposed on the area of the explanation subject. On the other hand, if the human body and the area of the explanation subject do not overlap, the transparency of the area of the explanatory material that overlaps with the human body area is changed, and the explanatory material is superimposed on the video information. This makes it possible to confirm the area of the explanation subject on the screen even if the human body overlaps the area of the explanation subject.
[0156] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. Although a number of exemplary embodiments have been described above in detail, the present invention can be implemented, for example, as a system, device, method, program, or recording medium (storage medium). Specifically, the present invention may be applied to a system consisting of a number of devices (for example, a host computer, an interface device, an imaging device, a WEB application, etc.), or may be applied to a device consisting of a single device. Also, for example, some or all of the functions of each functional unit shown in FIG. 1 may be included in a device different from the image superimposition device A1003. Specifically, a device or storage device different from the image superimposition device A1003 may have these functional units, and the functions of each embodiment may be realized by communicating with the image superimposition device A1003 based on a wired or wireless connection. Examples of the different device include the video capture device A1001, the document capture device A1002, an information processing device (not shown), or a server (not shown).
[0157] Also, for example, one or more functional units in Fig. 1 may be realized by one or more computers different from the image superimposition device A1003. Also, the image superimposition device A1003 may have functions equivalent to those of the video acquisition device A1001, the material acquisition device A1002, and the monitor device A1013. In this case, for example, the image superimposition device A1003 may be configured to acquire images and generate videos from the acquired images. Also, for example, the image superimposition device A1003 may be configured to acquire explanatory materials. Also, for example, the image superimposition device A1003 may be configured to display videos or images such as superimposed videos or superimposed images.
[0158] Also, some or all of the functions of the functional units in Fig. 1 may be provided by one or more devices other than the image superimposing device A 1003, or it is also possible for the image superimposing device A 1003 to have all of the functions in Fig. 1. The same applies to Figs. 11, 15, and 19. That is, the same can be applied to the image superimposing devices B 1003, C 1003, and D 1003.
[0159] The disclosure of this embodiment includes the following configuration, method, and program.
[0160] (Configuration 1) a first extraction means for extracting a region of a human body from an image; a superimposing means for superimposing predetermined superimposing information on the image; a first determination means for determining whether the human body is performing a predetermined motion and outputting a determination result; a second extraction means for extracting, based on the determination result, the region of the human body, and the predetermined superimposition information, a region of the predetermined superimposition information that overlaps with the region of the human body as an overlap region; a transparency changing unit that changes the transparency of at least a part of the predetermined overlay information to be higher in accordance with the overlapping area.
[0161] (Configuration 2) The processing device according to configuration 1, wherein the superimposition means, when the judgment result is that the human body is not performing the specified movement, superimposes the specified superimposition information with the transparency of the image unchanged.
[0162] (Configuration 3) The processing device according to configuration 1 or 2, wherein the second extraction means extracts the overlapping area from the image when the determination result is that the human body is performing the predetermined movement.
[0163] (Configuration 4) 4. The processing device according to any one of configurations 1 to 3, wherein the transparency changing means changes the transparency of the overlapping region.
[0164] (Configuration 5) an estimation means for detecting a human body from the image, estimating a skeleton of the detected human body, and outputting an estimation result; The processing device according to any one of configurations 1 to 4, wherein the first determination means determines whether or not the human body is performing the predetermined movement based on the estimation result estimated by the estimation means.
[0165] (Configuration 6) 6. The processing device according to configuration 5, wherein the second extraction means extracts a part of the human body as the overlapping region when extracting the overlapping region.
[0166] (Configuration 7) 7. The processing device according to any one of configurations 1 to 6, wherein the predetermined motion is at least a motion of moving an arm.
[0167] (Configuration 8) 8. The processing device according to any one of configurations 1 to 7, further comprising display control means for displaying the image or video superimposed by the superimposing means on a screen of a display device.
[0168] (Configuration 9) a second determination means for determining whether the human body is speaking based on voice information; The processing device according to any one of configurations 1 to 8, characterized in that, when it is determined that the human body is speaking, the second determination means determines that the human body is providing an explanation by voice.
[0169] (Configuration 10) The processing device described in configuration 9, characterized in that the second determination means outputs first information when it determines that the human body is speaking, and outputs second information when it determines that the human body is not speaking.
[0170] (Configuration 11) 11. The processing device according to configuration 10, further comprising a measuring means for measuring a time from the end of the speech of the human body based on the second information, and outputting the measured time.
[0171] (Configuration 12) The processing device according to configuration 11, characterized in that the transparency changing means continues to change the transparency of the specified superimposition information based on the measured time until a specified time has elapsed since the human body stopped speaking.
[0172] (Configuration 13) a third extraction means for extracting a predetermined keyword from each of the voice information and the predetermined superimposition information, The processing device according to configuration 11 or 12, characterized in that the third extraction means has a third determination means for determining whether the keywords extracted from the voice information and the specified overlay information respectively match or are similar.
[0173] (Configuration 14) The processing device described in configuration 13, characterized in that the third determination means outputs third information when it determines that the keywords extracted from the voice information and the specified overlay information, respectively, match or are similar, and outputs fourth information when it determines that the keywords extracted from the voice information and the specified overlay information, respectively, do not match or are similar.
[0174] (Configuration 15) 15. The processing device according to configuration 14, wherein the measuring means measures the time from when the human body stops speaking based on the second information and the fourth information, and outputs the measured time.
[0175] (Configuration 16) The processing device according to any one of configurations 1 to 15, further comprising an identification means for identifying an explanation area, which is an area in which the human body is explaining by the specified action or voice, from the specified overlay information.
[0176] (Configuration 17) 17. The processing device according to configuration 16, further comprising highlighting means for highlighting the description area.
[0177] (Configuration 18) A method for controlling a processing device, comprising the steps of: A first extraction step of extracting a region of a human body in an image; a superimposing step of superimposing predetermined superimposing information on the image; a determination step of determining whether the human body is performing a predetermined motion and outputting a determination result; a second extraction step of extracting, based on the determination result, the region of the human body, and the predetermined superimposition information, a region of the predetermined superimposition information that overlaps with the region of the human body as an overlapping region; and changing the transparency of at least a part of the predetermined overlay information so as to be increased in accordance with the overlapping area.
[0178] (Configuration 19) A program for causing a computer to execute a control method for a processing device, the program comprising: A first extraction step of extracting a region of a human body in an image; a superimposing step of superimposing predetermined superimposing information on the image; a determination step of determining whether the human body is performing a predetermined motion and outputting a determination result; a second extraction step of extracting, based on the determination result, the region of the human body, and the predetermined superimposition information, a region of the predetermined superimposition information that overlaps with the region of the human body as an overlapping region; and changing the transparency of at least a part of the predetermined overlay information so as to be increased in accordance with the overlapping area.
[0179] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]
[0180] A1000 Automatic Photography System A1001 Image acquisition device A1002 Data acquisition device A1003 Image overlay device A1004 Image acquisition unit A1005 Material acquisition department A1006 Skeleton information estimation unit A1007 Human body motion determination section A1008 Region division processing unit A1009 Overlapping area extraction part A1010 Transparency change section A1011 Image superimposition unit A1012 Video output device A1013 Monitor device
Claims
1. a first extraction means for extracting a region of a human body from an image; a superimposing means for superimposing predetermined superimposing information on the image; a first determination means for determining whether the human body is performing a predetermined motion and outputting a determination result; a second extraction means for extracting, based on the determination result, the region of the human body, and the predetermined superimposition information, a region of the predetermined superimposition information that overlaps with the region of the human body as an overlap region; a transparency changing unit that changes the transparency of at least a part of the predetermined overlay information to be higher in accordance with the overlapping area.
2. The processing device according to claim 1, characterized in that the superimposition means, when the judgment result is that the human body is not performing the specified movement, superimposes the specified superimposition information with the transparency of the image unchanged.
3. 2 . The processing device according to claim 1 , wherein the second extraction means extracts the overlapping area from the image when the determination result indicates that the human body is performing the predetermined movement.
4. The processing device according to claim 1 , wherein the transparency changing means changes the transparency of the overlapping region.
5. an estimation means for detecting a human body from the image, estimating a skeleton of the detected human body, and outputting an estimation result; 2 . The processing device according to claim 1 , wherein the first determining means determines whether or not the human body is performing the predetermined motion based on the estimation result obtained by the estimating means.
6. 6. The processing device according to claim 5, wherein the second extraction means extracts a part of the human body as the overlapping region when extracting the overlapping region.
7. The processing device according to claim 1 , wherein the predetermined motion is at least an arm movement.
8. 2. The processing device according to claim 1, further comprising display control means for displaying the image or video superimposed by said superimposing means on a screen of a display device.
9. a second determination means for determining whether the human body is speaking based on voice information; 2 . The processing device according to claim 1 , wherein the second determination means, when determining that the human body is speaking, determines that the human body is providing an explanation by voice.
10. 10. The processing device according to claim 9, wherein the second determination means outputs first information when it determines that the human body is speaking, and outputs second information when it determines that the human body is not speaking.
11. 11. The processing device according to claim 10, further comprising a measuring means for measuring a time from the end of the speech of the human body based on the second information, and outputting the measured time.
12. The processing device according to claim 11 , wherein the transparency changing means continues to change the transparency of the predetermined superimposition information until a predetermined time has elapsed since the human body stopped speaking, based on the measured time.
13. a third extraction means for extracting a predetermined keyword from each of the voice information and the predetermined superimposition information, 12. The processing device according to claim 11, further comprising a third determination means for determining whether the keywords extracted by the third extraction means from the voice information and the predetermined superimposition information respectively match or are similar.
14. The processing device according to claim 13, characterized in that the third determination means outputs third information when it determines that the keywords extracted from the voice information and the specified overlay information, respectively, match or are similar, and outputs fourth information when it determines that the keywords extracted from the voice information and the specified overlay information, respectively, do not match or are similar.
15. 15. The processing device according to claim 14, wherein the measuring means measures the time from when the human body stops speaking based on the second information and the fourth information, and outputs the measured time.
16. The processing device according to claim 1 , further comprising: a specifying unit for specifying an explanation area, which is an area where the human body is explaining by the predetermined motion or voice, from the predetermined superimposition information.
17. 17. The processing device according to claim 16, further comprising highlighting means for highlighting the description area.
18. A method for controlling a processing device, comprising the steps of: A first extraction step of extracting a region of a human body in an image; a superimposing step of superimposing predetermined superimposing information on the image; a determination step of determining whether the human body is performing a predetermined motion and outputting a determination result; a second extraction step of extracting, as an overlapping region, a region of the predetermined superimposition information that overlaps with the region of the human body based on the determination result, the region of the human body, and the predetermined superimposition information; and changing the transparency of at least a part of the predetermined overlay information so as to be increased in accordance with the overlapping area.
19. A program for causing a computer to execute a control method for a processing device, the program comprising: A first extraction step of extracting a region of a human body in an image; a superimposing step of superimposing predetermined superimposing information on the image; a determination step of determining whether the human body is performing a predetermined motion and outputting a determination result; a second extraction step of extracting, as an overlapping region, a region of the predetermined superimposition information that overlaps with the region of the human body based on the determination result, the region of the human body, and the predetermined superimposition information; and changing the transparency of at least a part of the predetermined overlay information so as to be increased in accordance with the overlapping area.
Citation Information
Patent Citations
Image display method, program, image display apparatus, and image display system
JP2005287004A
Image display and image display method therefor
JP2010039125A
Explanation support device, explanation support method, and explanation support program
JP2016194877A
Moving image output device, moving image output method, moving image output program, and moving image output system
JP2020155961A
Information processing system, information processing device and control program thereof
JP2023016269A
