Image processing system

Abstract

To provide an image processing system that, when photographing and recording a real subject and a virtual subject at the same time, can change whether to record the virtual subject for every virtual subject.SOLUTION: An image processing system has: an imaging unit that photographs a real object to create a real image; a virtual object information acquisition unit that acquires, from the outside, display information of a virtual object; a virtual object information changing unit that changes the display information of the virtual object on the basis of type information of the virtual object; a virtual object composition unit that performs composition of the real image and a virtual object image created by the virtual object information changing unit; and a recording unit that records a composite image obtained through the composition performed by the virtual object composition unit.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present disclosure relates to an image processing system.

Background Art

[0002] In recent years, technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) have been advancing. These technologies are realized by using head-mounted displays (HMDs), camera-equipped smartphones, tablet terminals, etc. By using the cameras of these devices, it is possible to combine and display or record a real subject in the captured image, which is a real subject, and a virtual subject, which is computer graphics representing a virtual object that does not exist at the same location as the real subject, in a single image. With this technology, by displaying people and objects existing in a distant place as virtual characters and virtual objects, it is possible to take pictures as if they were in the same space.

[0003] Separately from the above technology, imaging devices that can automatically perform framing, releasing, etc. and perform autonomous imaging without depending on the operator's operation of the photographer have become widespread. Security cameras are widely used products for photographing desired locations for crime prevention purposes. In recent years, the applications have diversified, and automatic imaging cameras for photographing moving people and objects, targeting more individual users, have also emerged.

[0004] In Patent Document 1, a user is made to recognize a virtual space and virtual objects using an HMD, and when a plurality of virtual objects overlap, an appropriate display is realized by moving or deleting one virtual object. Further, in Patent Document 2, when photographing and displaying a real object, from the viewpoint of privacy protection, a part of the image is masked and displayed.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] In an apparatus that displays an image obtained by synthesizing a photographed real space and a virtual object on a display such as an HMD, it is necessary to display the virtual object for the purpose of user viewing. However, in the case of photography, from the perspective of security, it may not be desirable to record the virtual object in the image. However, in the prior art, since processing such as erasing or masking is performed on the displayed image, the display is performed to view the subject, but it cannot be said that it is not left in the recorded image from the perspective of security and privacy protection. Thus, in the prior art, there has been a problem that when simultaneously photographing and recording a real subject and a virtual subject, the displayed image and the recorded image cannot be made different images.

[0007] Therefore, an object of the present disclosure is to provide an image processing system capable of changing whether or not to record each virtual object when simultaneously photographing and recording a real object and a virtual object.

Means for Solving the Problems

[0008] The first aspect of the present disclosure is an imaging unit that photographs a real object to generate a real image, a virtual object information acquisition unit that acquires display information of a virtual object from the outside, a virtual object information changing unit that changes the display information of the virtual object based on the type information of the virtual object, a virtual object synthesizing unit that synthesizes the real image and the virtual object image generated by the virtual object information changing unit, a recording unit that records the synthesized image synthesized by the virtual object synthesizing unit, and is characterized by having the above.

Advantages of the Invention

[0009] According to the present disclosure, when simultaneously photographing and recording a real subject and a virtual subject, it is possible to change whether to record for each virtual subject.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiments for Carrying Out the Invention

[0011] The present disclosure relates to an image processing system that displays and records a composite image obtained by synthesizing a virtual object and a real object. Here, the virtual object includes virtual characters. In the present disclosure, in order to clarify that the virtual object includes virtual characters, expressions such as "virtual characters and virtual objects" may be adopted. It should be noted that this description is not intended to limit that the "virtual object" does not include the "virtual characters".

[0012] <The First Embodiment> The first embodiment of the present invention will be described. The first embodiment is an image display system composed of a virtual object display device 200, an image processing device 300, and a server 400.

[0013] FIG. 2 is an external view of the virtual object display device 200 that synthesizes a virtual object with a real object and displays it to the user in the present embodiment. Hereinafter, the virtual object display device will also be simply referred to as the display device.

[0014] The virtual object display device 200 acquires display image information and display position information of a virtual object from an external server 400 via a communication unit (not shown). The virtual object display device 200 includes an imaging unit 201 and a display unit 202. The imaging unit 201 captures image information of the real space. The display unit 202 superimposes and displays an image of the virtual object on the image of the real space captured by the imaging unit 201 based on the acquired image information and display position information of the virtual object. By using the virtual object display device 200, the user can view an image in which the real space image and the virtual object image are simultaneously displayed. In this embodiment, the appearance of a typical glasses-type virtual object display device is shown as the virtual object display device 200, but virtual object display devices with other appearances may also be adopted.

[0015] FIG. 3 is an external view of an image processing device 300 that automatically records a real subject and a virtual subject simultaneously in this embodiment. The image processing device 300 can also be regarded as an imaging device that captures a real object and generates a real image.

[0016] The image processing device 300 captures an image of the real space via an imaging optical unit 101 and a sensor 102. The image processing device 300 can zoom via the imaging optical unit 101 and rotate the angle of view about at least one of the pan axis, tilt axis, and roll axis via a position and orientation control unit (not shown). The rotation about the pan axis and tilt axis is performed by adjusting the overall angle including the imaging optical unit 101 and the sensor 102. The rotation about the roll axis is performed by adjusting the angle of the sensor 102. Also, the image processing device 300 may be capable of driving in at least one of the up-down, left-right, and front-back directions. In this embodiment, the appearance of a typical automatic shooting camera is shown as the image processing device 300, but the image processing device 300 may be configured to be framable by moving and rotating the body itself like a drone. Also, the image processing device 300 may be configured to be framable by an external device such as a gimbal-like operating pan-tilt head.

[0017] FIG. 1 is a system configuration diagram of the image processing apparatus 300 according to the present embodiment. The image processing apparatus 300 includes an imaging optical unit 101, a sensor 102, an imaging processing unit 103, a data transfer control unit 104, a DRAM 105, a signal processing unit 106, a distance information acquisition unit 107, a position / attitude information acquisition unit 108, and a position / attitude control unit 109. The image processing apparatus 300 further includes a CPU 110, a ROM 111, a subject determination unit 112, a communication unit 113, a virtual object information change unit 114, a virtual object synthesis unit 115, a recording unit 116, and an automatic shooting control unit 117.

[0018] The imaging optical unit 101 includes a lens, a diaphragm, etc., and performs focus adjustment and exposure adjustment. The sensor 102 is a CMOS sensor or the like that converts an optical image into an electrical signal to capture an image, and converts the captured analog image signal into digital image data by an A / D conversion circuit and outputs it. The imaging processing unit 103 performs image processing such as noise removal on the digital image data output from the sensor 102. The data transfer control unit 104 controls writing / reading of data to / from the DRAM 105. The imaging optical unit 101, the sensor 102, and the imaging processing unit 103 can also be regarded as an imaging unit that captures a real object and generates a real image.

[0019] The signal processing unit 106 performs development processing for generating a YUV signal on the image data, JPEG processing for generating a JPEG image from the YUV signal, and the like.

[0020] The distance information acquisition unit 107 is composed of a distance information acquisition sensor or the like that measures the distance between the subject and the image processing apparatus 300. This distance information acquisition sensor is a sensor such as a phase difference sensor that detects the phase difference of light rays incident from the subject to measure the distance, or a distance sensor that emits light rays toward the subject and measures the distance by measuring the time until the reflected light returns. The distance information acquisition sensor may also be the same sensor as the sensor 102.

[0021] The position and orientation information acquisition unit 108 is a sensor that measures the position and orientation of the image processing apparatus 300. For acquiring the position information, a positioning system sensor using satellites is used, and for acquiring the orientation information, an acceleration sensor is used.

[0022] The CPU 110 controls various operations via a bus. The ROM 111 is a memory capable of recording information necessary for the image processing apparatus 300 to record an image. The communication unit 113 can communicate various information by wireless communication means such as WiFi, and can acquire virtual object information including display information of a virtual object from the outside.

[0023] The subject determination unit 112 determines whether or not the subject in the captured image is a subject set to transmit subject information to the outside. Specifically, the subject determination unit 112 compares an image of a subject pre-recorded in the ROM 111 according to a registration instruction from the user with the image captured by the sensor 102 to determine a subject in the real space. Further, the subject determination unit 112 can transmit the determined subject information to the outside via the communication unit 113. Also, the subject determination unit 112 can select the image most similar to the image of the virtual object acquired via the communication unit 113 from among a plurality of images pre-recorded in the ROM 111.

[0024] The virtual object information changing unit 114 is configured to be able to change the display information of the virtual object acquired via the communication unit 113. The virtual object information changing unit 114 can change virtual object information such as deleting the virtual object, changing it to an image pre-registered by the user in the ROM 111, and changing it to the image selected by the subject determination unit 112 as an image similar to the virtual object from among a plurality of pre-registered images. The virtual object information changing unit 114 changes the display information of the virtual object based on the type information of the virtual object described later.

[0025] The virtual object composition unit 115 is configured to be able to compose the virtual object image generated (changed) by the virtual object information changing unit 114 and the real image captured by the sensor 102. The recording unit 116 records the composite image composed by the virtual object composition unit 115 on a recording medium.

[0026] The image processing apparatus 300 in this embodiment can automatically photograph a subject. The automatic shooting control unit 117 controls the shooting angle of view and the recording timing for automatic shooting. The automatic shooting control unit 117 in this embodiment determines the orientation of the face of the imaged person. The automatic shooting control unit 117 excludes the image information of the virtual person and the virtual object to be deleted by the virtual object information changing unit 114, and controls the shooting angle of view and the recording timing based on the information of other subjects.

[0027] The CPU 110 controls the position and posture of the image processing apparatus 300 via the position and posture control unit 109 so that the orientation of the face of the imaged person is frontal based on the result determined by the automatic shooting control unit 117. Further, the automatic shooting control unit 117 detects the timing when the face of the imaged person becomes a smiling face and the timing when the difference between a plurality of continuously imaged images becomes large. The CPU 110 performs automatic shooting by instructing the sensor 102 and the recording unit 116 to capture and record an image at the timing detected by the automatic shooting control unit 117.

[0028] Each of the above-described processing units may be realized by a dedicated logic circuit or may be realized by a general-purpose processor executing a computer program.

[0029] FIG. 4 is a diagram for explaining the processing when the image processing apparatus 300 in this embodiment images the real space.

[0030] 401 is a shooting space represented in a global coordinate system with the center of the image processing apparatus 300 as the origin 402. 410 is a real person existing in the real space and using the virtual object display device 200. Also, 411, 412, and 413 are real objects existing in the real space. Note that the real person 410 is a kind of real object and may be treated as a real object hereinafter.

[0031] The image processing device 300 can transmit the information of the captured subject to an external server 400 using the communication unit 113. The server 400 can transmit the received image to a virtual object display device 200 at another location that does not exist in the shooting space 401, so that the image of the captured subject can be visually recognized by a person who does not exist in the shooting space 401.

[0032] Here, which subject information the image processing device 300 transmits to the server 400 can be set by registering an image in the ROM 111 of the image processing device 300 in advance. The image processing device 300 can select a subject to transmit information to the server 400 by comparing the image registered in the ROM 111 with the images of real people and real objects captured by the sensor 102 using the subject determination unit 112. In this embodiment, the information of the real object 410 and the real object 405 is set to be transmitted to the server 400, and the information of the real objects 411 to 413 is not transmitted to the server 400. This is because the real object 410 and the real object 405 are to be visually recognized by a person who does not exist in the shooting space 401, but the real objects 411 to 413 do not need to be visually recognized by a person who does not exist in the shooting space 401.

[0033] The subject information transmitted by the image processing device 300 to the server 400 includes display information (or image information) regarding the appearance for displaying the subject as a virtual object, the position information of the subject or virtual object, and the type information of the subject or virtual object. In the present disclosure, the subject information transmitted by the image processing device 300 to the server 400 is also referred to as virtual object information.

[0034] The display information of the subject is obtained by the image processing device 300 capturing the subject using the imaging unit as described above.

[0035] The position information of the subject can be calculated by the image processing device 300 using the distance information acquisition unit 107 and the position / attitude information acquisition unit 108. The position information of the subject includes, for example, the distance d from the subject with respect to the image processing device 300 and the azimuth θ of the subject.

[0036] The subject type information is information indicating how to record the subject when recording. When the image processing apparatus 300 captures a real person 410, which is the subject to be transmitted to the server 400, and transmits the image information and the position information to the server 400, the type information is also transmitted together. This type information can be registered when registering the subject target in the ROM 111 of the image processing apparatus. Details of the type information will be described later with reference to FIG. 6.

[0037] FIG. 5 is a diagram for explaining the process when the image processing apparatus 300 in the present embodiment synthesizes a real subject and a virtual subject. In the present embodiment, a case will be described where people at remote locations communicate as if they were in the same space, and the communication is imaged and recorded. Participants who participate in the virtual meeting room opened by the server 400 register the people and objects that should be visible to other participants in the image processing apparatus 300 at each remote location where the participants are located. The image processing apparatus 300 transmits the image information, the position information, and the type information of the registered object to the server 400.

[0038] Also, it is assumed that each participant is using the virtual object display device 200. Based on the received participant and object information, the server 400 transmits the image information, the position information, and the type information about the participants and objects to the image processing apparatus 300 at each remote location and the virtual object display device 200 used by each participant. At this time, the server 400 changes and transmits the respective position information so that the positions of each participant and object do not overlap. The virtual object display device 200 displays virtual people and virtual objects based on the image information and the position information of the people and objects received from the server 400. Each participant can visually recognize the virtual people and virtual objects by checking the display image of the virtual object display device 200. FIG. 5 shows a real person 410 visually recognizing virtual people and virtual objects via the virtual object display device 200. The real person 410 can visually recognize virtual people 500, 510, 511, 512, 520, 521 and virtual objects 501, 513.

[0039] The image processing apparatus 300 can synthesize a virtual person and a virtual object generated based on the image information (display information) and position information of a person and an object received from the server 400 with the image of the captured real space, and record the synthesized image.

[0040] FIG. 6 is a diagram for explaining the type information indicating how the image processing apparatus 300 in the present embodiment records virtual persons and virtual objects when recording. In the present embodiment, type information with type numbers 0 to 50 is defined. This type number is set for each virtual person and virtual object and is transmitted from the server 400 to the image processing apparatus 300. When the type number transmitted from the server 400 to the image processing apparatus 300 is 0, the image processing apparatus 300 does not change the image information of the virtual person and virtual object transmitted from the server 400. The image processing apparatus 300 synthesizes and records the images of the virtual person and virtual object with the subject in the real space. When the type number is 1, the image processing apparatus 300 does not record the virtual person and virtual object transmitted from the server 400. When the type number is 2, the subject determination unit 112 of the image processing apparatus 300 selects the image most similar to the images of the virtual person and object transmitted from the server 400 from among a plurality of images pre-recorded in the ROM 111, and changes and records it to the selected image. Specifically, the image processing apparatus 300 changes the images of the virtual person and object transmitted from the server 400 to the image selected by the subject determination unit 112, and synthesizes and records it with the subject in the real space. When the type number is 3 to 50, the image processing apparatus 300 changes the images of the virtual person and object transmitted from the server 400 to the images pre-registered by the user, and synthesizes and records it with the subject in the real space. The user can pre-register 48 images corresponding to type numbers 3 to 50 in the ROM 111 of the image processing apparatus 300. The image processing apparatus 300 changes the images of the virtual person and object transmitted from the server 400 to the images corresponding to the type number transmitted from the server 400, and synthesizes and records it with the subject in the real space.

[0041] The category information may be set according to the security level of the virtual object, or may be set according to whether the virtual object is the main subject. For example, the user sets whether it is the security level or the main subject, and the server 400 may determine the category information (category number) according to whether it is the security level or the main subject.

[0042] FIG. 7 is a diagram for explaining the process when the image processing apparatus 300 in the present embodiment changes the virtual subject image according to the category information and records the real subject and the virtual subject at the same time. The image processing apparatus 300 changes and records the image information of each subject based on the image information, position information, and category information of each subject transmitted from the server 400.

[0043] Regarding the real person 410, the virtual person 500, and the real object 405, the category number 0 (record as it is) is transmitted as the category information from the server 400. Therefore, the image processing apparatus 3 00 records by synthesizing the image data of the real space of the real objects 411, 412, 413, etc. captured by the sensor 102 without changing the image information of the real person 410, the virtual person 500, and the real object 405 transmitted from the server 400.

[0044] Regarding the virtual object 501, the category number 1 (do not record) is transmitted as the category information from the server 400. This is because although the virtual object 501 is to be made visible to the real person 410 and the virtual person 500, it is not desired to record it for security reasons. Therefore, as shown by 700 in FIG. 7, the image processing apparatus 300 does not record the virtual object 501.

[0045] Regarding the virtual persons 511 and 512, the category number 0 (record as it is) is transmitted as the category information from the server 400. Therefore, the image processing apparatus 300 records by synthesizing the image data of the real space captured by the sensor 102 without changing the image information of the virtual persons 511 and 512 transmitted from the server 400.

[0046] Regarding the virtual character 510, the type number 3 (changed to a similar image and stored) is sent as type information from the server 400. This is because, while we want to make the virtual character 510 refer to other characters such as virtual character 511 and virtual character 512, for privacy protection, we do not want to record it with the original image. The image of the virtual character 710 corresponding to the type number 3 is pre-recorded in the ROM 111. The image processing device 300 changes the image of the virtual character 510 sent from the server 400 to the virtual character 710, and combines it with the image data of the real space captured by the sensor 102 for recording.

[0047] Regarding the monitor which is the virtual object 513, the type number 4 (changed to a registered image and stored) is sent as type information from the server 400. This is because, while we want to make the screen of the monitor which is the virtual object 513 refer to other characters such as virtual characters 510 - 512, for security reasons, we do not want to record the image of the monitor screen as it is. 711 is a virtual object corresponding to the type number 4 and is an image of a monitor with the screen in a non-display state. Since the image of this virtual object 711 is pre-recorded in the ROM 111, the image processing device 300 changes the image of the virtual object 513 sent from the server 400 to the virtual object 711, and combines it with the image data of the real space captured by the sensor 102 for recording.

[0048] Regarding virtual characters 520 and 521, type number 2 (recorded after changing to a similar image) is sent as type information from server 400. This is because, while we want to make virtual characters 520 and 521 visible to other virtual characters 410, 500, 510, 511, and 512, for privacy protection reasons, we do not want to record them as their original images. When the type number is 2, the subject determination unit 112 of the image processing device 300 selects, from among a plurality of images pre-recorded in the ROM 111, the images that are most similar to the images of virtual characters 520 and 521 sent from server 400. In this embodiment, similar image 720 of virtual character 520 and similar image 721 of virtual character 521 have been selected. The image processing device 300 changes the images of virtual characters 520 and 521 sent from server 400 to the selected images 720 and 721, and synthesizes and records them with the image data of the real space captured by sensor 102.

[0049] Figure 8 shows a shooting sequence for simultaneously recording a real subject and a virtual subject in this embodiment.

[0050] When the CPU 110 of the image processing device 300 is instructed by the user to take a picture (S801), it captures the real space and determines the subject (S802). This subject determination is performed by the subject determination unit 112 of the image processing device 300 comparing the image of the subject pre-registered and recorded in the ROM 111 upon instruction from the user with the image captured by the sensor 102. The CPU 110 of the image processing device 300 calculates the position information of the subject using the distance information acquisition unit 107 and the position / pose information acquisition unit 108 for the determined subject (S803). The CPU 110 of the image processing device 300 transmits the information (image information, position information, type information) of the subject in the captured real space to the server 400 using the communication unit 113 (S804). Note that the type information for determining how to record the subject is pre-registered by the user and stored in the ROM 111.

[0051] The CPU 110 of the image processing apparatus 300 receives the information (image information, position information, type information) of the subjects of virtual characters and virtual objects transmitted from the server 400 using the communication unit 113 (S805). The communication unit 113 corresponds to a virtual object information acquisition unit that acquires the display information of the virtual object from the outside. The CPU 110 stores the acquired subject information in the DRAM 105.

[0052] The CPU 110 of the image processing apparatus 300 changes the image information of the virtual characters and virtual objects in the virtual object information changing unit 114 according to the type information acquired from the server 400 (S806). The virtual object information changing unit 114 changes the display information of the virtual object based on the type information of the virtual object.

[0053] The CPU 110 of the image processing apparatus 300 synthesizes the image of the real space captured by the sensor 102 and the images of the virtual characters and virtual objects generated by the virtual object information changing unit 114 in the virtual object synthesizing unit 115 (S807). The virtual object synthesizing unit 115 synthesizes the real image and the virtual object image generated by the virtual object information changing unit.

[0054] The CPU 110 of the image processing apparatus 300 determines whether it is the timing to record the synthesized image in the automatic shooting control unit 117 (S808). If it is the recording timing, the CPU 110 records the synthesized image in the recording unit 116 (S809). When the CPU 110 of the image processing apparatus 300 is instructed by the user to end shooting (S810), it stops shooting the real space (S811).

[0055] According to the present embodiment, by the above control, all virtual characters and virtual objects can be visually recognized and referred to by the participants, and virtual characters and virtual objects that are problematic from the viewpoints of security and privacy protection can have their images changed and recorded.

[0056] <Modification Example of the First Embodiment> In addition, in the present embodiment, information on virtual characters and virtual objects is transmitted and received via the server 400. However, the image processing apparatus 300 and the virtual object display apparatuses 200 at each remote location may directly communicate with each other.

[0057] Also, in this embodiment, the image processing apparatus 300 determines the subject in the captured real space and transmits the subject information to the server 400. However, an imaging device different from the image processing apparatus 300 may determine the subject and transmit the subject information to the server 400.

[0058] Also, in this embodiment, when the image processing apparatus 300 changes the recorded images of the virtual person and the virtual object, the images are changed to the pre-registered images or similar images. However, the recorded images may be changed by performing mosaic processing, blindfold processing, or the like.

[0059] Also, in this embodiment, one type of composite image in which the images of the virtual person and the virtual object are changed according to the type information is recorded. However, a plurality of types of composite images generated according to a plurality of types of type information may be recorded. Specifically, a plurality of type information can be set for the virtual person and the virtual object. Then, the image processing apparatus 300 may change the display information of the virtual object according to each of the plurality of type information, and generate and record a plurality of composite images obtained by synthesizing the real image and the changed virtual object image respectively. For example, in this embodiment, only one type of type information with type number 1 can be registered in the ROM 111 in advance for the virtual object 501 in FIG. 5. However, it is also possible to register two types of type information with type numbers 0 and 1 in the ROM 111. When these two types of type information are registered, the image processing apparatus 300 records two types of composite images: a composite image in which the image of the virtual object 501 is not recorded, and a composite image that is recorded as it is without being changed. Thereby, it becomes possible to provide images with different security levels according to the user who refers to the recorded images. In this embodiment, only one type of type information with type number 1 can be registered in the ROM 111 in advance. However, it is also possible to register two types of type information with type numbers 0 and 1 in the ROM 111. When these two types of type information are registered, the image processing apparatus 300 records two types of composite images: a composite image in which the image of the virtual object 501 is not recorded, and a composite image that is recorded as it is without being changed. Thereby, it becomes possible to provide images with different security levels according to the user who refers to the recorded images.

[0060] Also, the automatic shooting control unit 117 in this embodiment may perform shooting so that the subject has an optimal composition and timing, excluding the information of the virtual person and the virtual object that are not recorded.

[0061] In addition, in this embodiment, although the image processing apparatus 300 performs the processing shown in the flowchart of FIG. 8, these processes may be executed for the entire image processing system. For example, some of the processes may be performed by the server 400. For example, the server 400 may change the display image information based on the type information of the virtual object, and transmit the changed display image information from the server 400 to the image processing apparatus 300. Note that, for a virtual object for which the type information is set to "not record", the server 400 does not have to transmit the information of the virtual object to the image processing apparatus 300. The image processing apparatus 300 may generate a composite image using the display image information of the virtual object transmitted from the server 400. Even in this case, the same effect as that of the first embodiment can be obtained.

[0062] <Second Embodiment> A second embodiment of the present disclosure will be described.

[0063] In this embodiment, the difference from the first embodiment is that the type information of the virtual person and the virtual object is generated not by receiving from the outside but within the image processing apparatus 300. Since the external view of the image processing apparatus 300, the external view of the virtual object display apparatus 200, the process of the image processing apparatus 300 capturing the real space, and the process when the image processing apparatus 300 synthesizes the real subject and the virtual subject are the same as those in the first embodiment, the description thereof will be omitted.

[0064] FIG. 9 is a system configuration diagram of the image processing apparatus 300 in this embodiment. The difference from the first embodiment is that the type information of the virtual object is generated not by receiving from the outside but by the virtual object type information generation unit 900. Hereinafter, the configuration different from the first embodiment will be mainly described.

[0065] In this embodiment, the virtual object type information generation unit 900 generates type information so as to record only the main subject of the virtual person as it is in the image, and change and record the images of the other virtual persons. The image processing apparatus 300 can discriminate the main subject by comparing the image registered in advance in the ROM 111 by the user as the main subject with the image of the virtual person.

[0066] In addition, the virtual object type information generation unit 900 generates type information such that a virtual object with a high security level changes and records an image, and a virtual object with a low security level records the image as it is. The image processing apparatus 300 can determine a virtual object with a high security level by comparing the image of the virtual object with the image pre-registered in the ROM 111 as an object with a high security level.

[0067] The virtual object information changing unit 114 changes the image information of the virtual object based on the type information generated by the virtual object type information generation unit 900. Similar to the first embodiment, the virtual object information changing unit 114 can change virtual object information such as deleting the virtual object, changing it to an image pre-registered in the ROM 111 by the user, or changing it to an image similar to the virtual object selected by the subject determination unit 112.

[0068] The virtual object combining unit 115 combines the virtual object image output from the virtual object information changing unit 114 and the real image captured by the sensor 102. The recording unit 116 records the image combined by the virtual object combining unit 115 on a recording medium.

[0069] FIG. 10 is a diagram for explaining the process when the image processing apparatus 300 changes the virtual subject image according to the type information and records the real subject and the virtual subject at the same time in the present embodiment. The image information and position information of each subject are transmitted from the server 400 to the image processing apparatus 300 in the same manner as in FIG. 5. The type information of each subject is generated by the virtual object type information generation unit 900. In the present embodiment, the image processing apparatus 300 can determine the main subject by comparing the image pre-registered in the ROM 111 by the user and the image of the virtual person as the main subject, and determines that only the real person 410 is the main subject.

[0070] The image processing apparatus 300 records the main subject 410 with the original image, and changes and records the other virtual persons 1000, 1010, 1011, 1012, 1020, and 1021 to similar images selected by the subject determination unit 112 in the same manner as in the first embodiment.

[0071] The image processing apparatus 300 can determine a virtual object with a high security level by comparing a plurality of images registered in advance in the ROM 111 as objects with a high security level with the image of the virtual object. In the present embodiment, the image processing apparatus 300 determines that the virtual objects 1001 and 1013 are objects with a high security level, and changes and records them as images different from the virtual objects 501 and 513 in FIG. 5. The image after this change is a similar image selected by the subject determination unit 112, similar to the first embodiment.

[0072] FIG. 11 is a shooting sequence for simultaneously recording a real subject and a virtual subject in the present embodiment. When the CPU 110 of the image processing apparatus 300 is instructed by the user to take a picture (S1101), it images the real space and determines the subject (S1102). This subject determination is performed by the subject determination unit 112 of the image processing apparatus 300 comparing the image of the subject registered in the ROM 111 in advance upon receiving an instruction from the user with the image captured by the sensor 102. The CPU 110 of the image processing apparatus 300 calculates the position information of the subject using the distance information acquisition unit 107 and the position / attitude information acquisition unit 108 for the determined subject (S1103). The CPU 110 of the image processing apparatus 300 transmits the information (image information, position information) of the subject in the imaged real space to the server 400 using the communication unit 113 (S1104).

[0073] The CPU 110 of the image processing apparatus 300 receives the information (image information, position information) of the virtual person and virtual object subjects transmitted from the server 400 using the communication unit 113 (S1105). The communication unit 113 corresponds to a virtual object information acquisition unit that acquires the display information of the virtual object from the outside. The CPU 110 stores the acquired subject information in the DRAM 105.

[0074] The CPU 110 of the image processing apparatus 300 generates the type information of each subject in the virtual object type information generation unit 900 from the subject image (display information) acquired from the server 400 (S1106).

[0075] The CPU 110 of the image processing apparatus 300 changes the image information of the virtual person and the virtual object in the virtual object information changing unit 114 according to the generated type information (S1107). The virtual object information changing unit 114 changes the display information of the virtual object based on the type information of the virtual object.

[0076] The CPU 110 of the image processing apparatus 300 combines the image of the real space captured by the sensor 102 and the images of the virtual person and the virtual object generated by the virtual object information changing unit 114 in the virtual object combining unit 115 (S1108). The virtual object combining unit 115 combines the real image and the virtual object image generated by the virtual object information changing unit.

[0077] The CPU 110 of the image processing apparatus 300 determines whether it is the timing to record the combined image in the automatic shooting control unit 117 (S1109). If it is the recording timing, the combined image is recorded in the recording unit 116 (S1110). When the CPU 110 of the image processing apparatus 300 is instructed by the user to end shooting (S1111), the imaging of the real space is stopped (S1112).

[0078] According to the present embodiment, by the above control, all virtual persons and virtual objects can be visually recognized and referred to by the participants, and virtual persons and virtual objects that have problems from the viewpoints of security and privacy protection can have their images changed and recorded. In addition, since the determination of the virtual persons and virtual objects whose image information is changed and recorded is executed by the image processing apparatus that performs the recording, the determination process is not required in the image processing apparatus that transmits the image information.

[0079] <Modification of the Second Embodiment> In addition, in this embodiment, the information of the virtual person and the virtual object is transmitted and received via the server 400, but the image processing apparatus 300 and the virtual object display apparatuses 200 at each remote location may communicate directly.

[0080] Also, in this embodiment, the image processing apparatus 300 determines the subject in the captured real space and transmits the subject information to the server 400. However, an imaging device different from the image processing apparatus 300 may determine the subject and transmit the subject information to the server 400.

[0081] Also, in this embodiment, when the image processing apparatus 300 changes the recorded images of the virtual person and the virtual object, it changes them to the pre-registered images or similar images. However, the recorded images may be changed by performing mosaic processing, blindfold processing, or the like.

[0082] Also, in this embodiment, one type of composite image in which the images of the virtual person and the virtual object are changed according to the type information is recorded. However, a plurality of types of composite images generated according to a plurality of types of type information may be recorded. Specifically, a plurality of types of type information can be set for the virtual person and the virtual object, and the image processing apparatus 300 may record composite images corresponding to the respective type information. For example, in this embodiment, the virtual object 501 in FIG. 5 can be registered in the ROM 111 as a subject image of type information with a high security level. However, it may be registered in the ROM 111 as a subject image having two types of type information, a high security level and a low security level. When these two types of type information are registered, the image processing apparatus 300 records two types of composite images: a composite image in which the image of the virtual object 501 is changed and a composite image that is recorded as it is without being changed. This makes it possible to provide images with different security levels according to the user who refers to the recorded images.

[0083] Also, the automatic shooting control unit 117 in this embodiment may perform shooting so that the subject has an optimal composition and timing, excluding the information of the virtual person and the virtual object that are not recorded.

[0084] In addition, in the present embodiment, the processing shown in the flowchart of FIG. 11 is performed by the image processing apparatus 300, but these processes may be executed for the entire image processing system. For example, some of the processes may be performed by the server 400. For example, the server 400 may generate the type information of the virtual object and transmit the virtual object information including the generated type information from the server 400 to the image processing apparatus 300. Furthermore, the change of the display image information based on the type information of the virtual object may also be performed by the server 400, and the changed display image information may be transmitted from the server 400 to the image processing apparatus 300. For the virtual object for which the type information is set to "not record" the server 400 does not have to transmit the information of the virtual object to the image processing apparatus 300. The image processing apparatus 300 may generate a composite image using the display image information of the virtual object transmitted from the server 400. Even in this way, the same effect as in the second embodiment can be obtained.

[0085] <Third Embodiment> A third embodiment of the present disclosure will be described.

[0086] In the present embodiment, the recording priority of each subject is included in the type information, and it is different from the first embodiment in that the display transparency of the virtual person and the virtual object determined as the shielding object is changed based on the recording priority. Since the external view of the image processing apparatus 300, the external view of the virtual object display apparatus 200, the process of the image processing apparatus 300 imaging the real space, and the process when the image processing apparatus 300 synthesizes the real subject and the virtual subject are the same as those in the first embodiment, the description thereof will be omitted.

[0087] FIG. 12 is a system configuration diagram of the image processing apparatus 300 in the present embodiment. The image processing apparatus 300 includes a subject occlusion determination unit 1200. The subject occlusion determination unit 1200 determines whether an object with a lower recording priority occludes an object with a higher recording priority based on the recording priority and position information of a real object or a virtual object. An object that occludes other objects is referred to as an occlusion object. The subject occlusion determination unit 1200 determines whether each subject is an occlusion object based on the information (recording priority, subject position information) transmitted from the server 400. The virtual object information change unit 114 generates a composite image by changing the display transparency of part or all of the image of a non-deleted virtual object based on the determination process of the subject occlusion determination unit 1200, and records the composite image. A non-deleted object is a virtual object whose type information is other than "do not record", and includes a virtual object to be recorded as it is and a virtual object whose image is changed and recorded. The change in display transparency is typically a process of lowering the transparency, and includes both the case of setting partial transparency and the case of setting complete transparency (i.e., deletion). Since the other configurations are the same as those in the first embodiment, the description thereof is omitted.

[0088] FIG. 13 is a diagram for explaining the recording priority indicating how the image processing apparatus 300 in the present embodiment records a virtual person or object when recording. In the present embodiment, recording priorities "high" to "low" are defined (FIG. 13(a)). In the present embodiment, the recording priority is set for each subject and transmitted from the server 400 to the image processing apparatus 300. In the present embodiment, the recording priority of each subject is registered in advance in the ROM 111 by the user.

[0089] The image processing apparatus 300 changes and records the display transparency of virtual characters so that a subject with a higher recording priority transmitted from the server 400 can be preferably recorded without being blocked by other subjects. The image processing apparatus 300 sets the recording priority of the main subject to the highest level. The image processing apparatus 300 can determine the main subject by comparing the image pre-registered in the ROM 111 by the user and the image of the virtual character as the main subject. The image processing apparatus 300 sets the recording priority of the subject determined as the main subject to a high level.

[0090] As shown in Fig. 13(b), consider a situation where virtual characters 1300b and 1300c with "medium" and "low" recording priorities are arranged in front of a real person 1300a with a "high" recording priority so as to block it. In this case, the image processing apparatus 300 can determine, using the position information (distance d, angle θ) of each subject, that the virtual characters 1300b and 1300c are arranged in front of the real person 1300a so as to block it, and thus determines the virtual characters 1300b and 1300c as occluders. The image processing apparatus 300 can preferably photograph the real person 1300a as shown in Fig. 13(c) by performing a transparency process on 1300b and 1300c. Note that it is not necessary to make all of the virtual objects determined as occluders transparent, and the occluder may be partially transparent so that at least a part (for example, the face, etc.) of an object with a higher priority behind the occluder can be recognized.

[0091] As shown in Fig. 13(d), consider a situation where a virtual character 1300c with a "low" recording priority is arranged in front of a virtual character 1300b with a "medium" recording priority so as to block it. Also in this case, similarly to the above, the image processing apparatus 300 determines the virtual character 1300c as an occluder using the position information (distance d, angle θ) of each subject. The image processing apparatus 300 can preferably photograph the virtual character 1300b as shown in Fig. 13(e) by performing a transparency process on 1300c. A method for determining an occluder using the position information (distance d, angle θ) will be described later.

[0092] FIG. 14 is a diagram for explaining the process when the image processing apparatus 300 changes the display transparency of virtual characters and virtual objects according to the determination of the subject occlusion determination unit 1200 and records a real subject and a virtual subject at the same time in the present embodiment. Similar to the first embodiment, the server 400 changes the respective position information so that the positions of the virtual characters and virtual objects do not overlap, and transmits the initial positions as the respective position information to the image processing apparatus 300 and the virtual object display apparatus 200 at each remote location. In the present embodiment, each virtual character moves from the initial position, preventing the subject to be recorded from being occluded by other virtual characters or objects and being preferably recorded. The difference from the first embodiment is that by changing the display transparency of the virtual characters and objects determined by the subject occlusion determination unit 1200 to be occluding objects, the subject with a higher recording priority is recorded so as not to be blocked by the occluding object. The image information and position information of the subject are transmitted from the server 400 to the image processing apparatus 300 in the same manner as in the first embodiment. In the present embodiment, the image processing apparatus 300 discriminates only the real person 410 as the main subject, and further sets the recording priority of the real person 410 to the highest "high".

[0093] In FIG. 14, the virtual character 1400 is assigned the type number 0, the virtual object 1401 is assigned the type number 1, the virtual characters 1420 and 1421 are assigned the type number 2, and the virtual characters 1410, 1411, and 1412 are assigned the type number 3. Similar to the first embodiment, the image processing apparatus 300 in the present embodiment processes virtual objects and virtual characters for each type number (FIG. 6), and changes the images of the virtual characters and objects according to the recording form. In the present embodiment, further, the subject occlusion determination unit 1200 determines virtual characters and objects that are occluding objects using the position information (distance d and angle θ) of each subject transmitted from the server 400.

[0094] The determination method of the subject occlusion determination unit 1200 will be described. The position information (distance dh and angle θh) of the subject with a higher recording priority and the position information (distance dl and angle θl) of the subject with a lower recording priority transmitted from the server 400 are compared. The subject occlusion determination unit 1200 determines that the subject with a lower recording priority is an occluding object when both of the following conditions 1 and 2 are satisfied.

[0095] (Condition 1) The subject with a lower recording priority is closer to the image processing apparatus 300. That is, dh > dl ··· (Equation 1) is satisfied.

[0096] (Condition 2) The difference in either the elevation angle θ(e) or the azimuth angle θ(a) of each subject with respect to the image processing apparatus 300 is less than the threshold value θth. That is, |θh(e) - θl(e)| < θth(e) ··· (Equation 2a) |θh(a) - θl(a)| < θth(a) ··· (Equation 2b) satisfies either one of them.

[0097] A plurality of threshold values θth are registered in advance in the ROM 111, and the subject occlusion determination unit changes the value of the threshold value θth to be adopted according to the distance dh of the subject with a high recording priority. The smaller the threshold value θth is adopted as dh is larger.

[0098] In FIG. 14, the virtual person 1400 with the recording priority of "medium" is closer to the real person 410 with the recording priority of "high" in terms of distance d (Equation 1), and the difference in the azimuth angle θ(a) is smaller than the threshold value (Equation 2b ). Therefore, the subject occlusion determination unit 1200 determines the virtual person 1400 as an occluding object. Thus, the virtual object information changing unit 114 makes the azimuth angle portion of the virtual person 1400 whose difference from the azimuth angle of the real person 410 is equal to or less than the threshold value transparent.

[0099] In FIG. 14, the virtual persons 1410 and 1411 with the recording priorities of "medium" and "low" respectively are closer to the virtual person 1412 with the recording priority of "high" in terms of distance d (Equation 1), and the difference in the elevation angle θ(e) is smaller than the threshold value (Equation 2a). Therefore, the subject occlusion determination unit 1200 determines the virtual persons 1410 and 1411 as occluding objects. The virtual object information changing unit 114 makes the elevation angle portion of the virtual persons 1410 and 1411 whose difference from the elevation angle of the virtual person 1412 is equal to or less than the threshold value transparent.

[0100] The image processing apparatus 300 records an image that has been subjected to a transparency process by the virtual object information changing unit 114.

[0101] FIG. 15 is a shooting sequence for simultaneously recording a real subject and a virtual subject in the present embodiment. When the CPU 110 of the image processing apparatus 300 is instructed to shoot by the user (S1501), it images the real space and determines the subject (S1502). This subject determination is performed by the subject determination unit 112 of the image processing apparatus 300 comparing the image of the subject registered in advance in the ROM 111 upon instruction from the user with the image captured by the sensor 102. The CPU 110 of the image processing apparatus 300 calculates the position information of the subject using the distance information acquisition unit 107 and the position / attitude information acquisition unit 108 for the determined subject (S1503). The CPU 110 of the image processing apparatus 300 transmits the information of the subject (image information, position information, type number, type information including recording priority) of the captured real space subject to the server 400 using the communication unit 113 (S1504). Note that the type information and recording priority for determining how to record the subject are registered in advance by the user and stored in the ROM 111.

[0102] The CPU 110 of the image processing apparatus 300 receives the information of the virtual person and virtual object subjects (image information, position information, type number, type information including recording priority) transmitted from the server 400 using the communication unit 113 (S1505). The communication unit 113 corresponds to a virtual object information acquisition unit that acquires display information of the virtual object from the outside. The CPU 110 stores the acquired subject information in the DRAM 105 (S1505).

[0103] The CPU 110 of the image processing apparatus 300 changes the image information of the virtual person and virtual object in the virtual object information changing unit 114 according to the type number acquired in S1505 (S1506). The virtual object information changing unit 114 changes the display information of the virtual object based on the type information of the virtual object.

[0104] The CPU 110 of the image processing apparatus 300 changes the display transparency of virtual characters and virtual objects in the virtual object information changing unit 114 according to the position information obtained in S1505 and the recording priority (S1507). The subject occlusion determination unit 1200 determines whether an object with a lower recording priority occludes an object with a higher recording priority based on the recording priorities and position information of real objects and virtual objects. The virtual object information changing unit 114 changes the display transparency of part or all of the image of the occluding object that occludes an object with a higher recording priority.

[0105] The CPU 110 of the image processing apparatus 300 synthesizes the image of the real space captured by the sensor 102 and the images of virtual characters and virtual objects generated by the virtual object information changing unit 114 in the virtual object synthesizing unit 115 (S1508). The CPU 110 of the image processing apparatus 300 determines whether it is the timing to record the synthesized image in the automatic shooting control unit 117 (S1509). If it is the recording timing, the synthesized image is recorded by the recording unit 116 (S1510). The CPU 110 of the image processing apparatus 300 stops imaging the real space when instructed by the user to end shooting (S1511) (S1512).

[0106] According to the present embodiment, by the above control, all virtual characters and virtual objects can be visually recognized and referred to by the participants, and virtual characters and virtual objects that are problematic from the viewpoints of security and privacy protection can have their images changed and recorded. Also, virtual characters and virtual objects that are problematic from the viewpoints of security and privacy protection want to record while changing their images The subject can be preferably recorded without being blocked by other virtual characters and virtual objects.

[0107] <Modification Example of the Third Embodiment> In this embodiment, information on virtual characters and virtual objects is transmitted and received via the server 400. However, the image processing apparatus 300 and the virtual object display apparatuses 200 at each remote location may communicate directly.

[0108] In addition, in this embodiment, the image processing apparatus 300 determines the subject in the captured real space and transmits the subject information to the server 400. However, an imaging device different from the image processing apparatus 300 may determine the subject and transmit the subject information to the server 400.

[0109] In addition, in this embodiment, when the image processing apparatus 300 changes the recorded images of virtual characters and virtual objects, the images are changed to pre-registered images or similar images. However, the recorded images may be changed by performing mosaic processing, covering the eyes, or the like.

[0110] In addition, the automatic shooting control unit 117 in this embodiment may perform shooting so that the subject has an optimal composition and timing, excluding the information of virtual characters and virtual objects that are not recorded.

[0111] In addition, in this embodiment, the recording priorities of the subjects are classified into three types: low, medium, and high. However, the types of recording priorities are not limited to three. For example, the recording priority may be indicated by a numerical value, and the subject with a higher numerical value may be determined as the subject with a higher recording priority.

[0112] In addition, in this embodiment, a part or all of the virtual characters are made transparent. However, the degree of occlusion may be evaluated in multiple stages, and the transparency of the virtual object image may be set to a value corresponding to the degree of occlusion. For this purpose, a plurality of thresholds for evaluating the degree of occlusion are set in advance. For example, as shown in FIG. 16, three-stage thresholds are set for the angle threshold θth used in Equation 2 as θth_a, θth_b, θth_c (θth_a < θth_b < θth_c). Then, the virtual object information changing unit 114 may set the transparency of the virtual characters included in the range of a smaller threshold to be larger (more transparent). In the example of FIG. 16, the transparency of the virtual character 1400a whose angle difference from the real person 410 is equal to or less than the threshold θth_a is set to 100%. Similarly, the transparency of the virtual character 1400b whose angle difference is between the thresholds θth_a and θth_b is 50%, and the transparency of the virtual character 1400c whose angle difference is between the thresholds θth_b and θth_c is 25%. Note that the angle threshold and the transparency value are not limited to the above, and may be set as appropriate.

[0113] Also, in this embodiment, when making part or all of the virtual character transparent, it may be set as a transparency ratio according to the recording priority of the virtual character. Specifically, the lower the recording priority, the higher the transparency, and the higher the recording priority, the lower the transparency may be set.

[0114] Also, in this embodiment, the recording priority is pre-registered by the user, but it may also be generated within the image processing apparatus 300. At this time, the image processing apparatus 300 has the same structure as that of the second embodiment, and the virtual object type information generation unit 900 generates the recording priority based on the type number. For example, for type number 0, the priority "high" is generated, for type number 2, the priority "medium" is generated, and for type numbers 3 and later, the priority "low" is generated.

[0115] Also, in this embodiment, the processes shown in the flowchart of FIG. 15 are performed by the image processing apparatus 300, but these processes may be executed for the entire image processing system. For example, some of the processes may be performed by the server 400. For example, the server 400 may change the display image information based on the type information of the virtual object, and transmit the changed display image information from the server 400 to the image processing apparatus 300. Furthermore, the server 400 may also change the display image information based on the type information of the virtual object, and transmit the changed display image information from the server 400 to the image processing apparatus 300. Also, the image information of the subject whose display transparency is changed based on the position information and recording priority of the virtual object may be transmitted from the server 400 to the image processing apparatus 300. Note that for a virtual object whose type information is set to "do not record", the server 400 does not need to transmit the information of the virtual object to the image processing apparatus 300. The image processing apparatus 300 may generate a composite image using the display image information of the virtual object transmitted from the server 400. Even in this way, the same effects as those of the third embodiment can be obtained.

[0116] <Other Embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0117] <Appendix> The present disclosure includes the following configurations. [Configuration 1] An imaging unit that captures a real object and generates a real image, A virtual object information acquisition unit that acquires display information of a virtual object from the outside, A virtual object information change unit that changes the display information of the virtual object based on the type information of the virtual object, A virtual object synthesis unit that synthesizes the real image and the virtual object image generated by the virtual object information change unit, A recording unit that records the composite image synthesized by the virtual object synthesis unit, An image processing system having the above. [Configuration 2] The type information of the virtual object is acquired by the virtual object information acquisition unit from the outside, The image processing system according to Configuration 1, characterized in that. [Configuration 3] Further comprising a type information generation unit that generates type information of the virtual object based on the display information, The image processing system according to Configuration 1 or 2, characterized in that. [Configuration 4] A plurality of the type information is set for the virtual object, The virtual object information change unit changes the display information of the virtual object according to each of the plurality of type information, The virtual object synthesis unit generates a plurality of composite images obtained by synthesizing the real image and each of the plurality of virtual object images generated by the virtual object information change unit, The recording unit records each of the plurality of composite images, The image processing system according to any one of Configurations 1 to 3, characterized in that... [Configuration 5] The type information is information representing the security level of the virtual object. The image processing system according to any one of Configurations 1 to 4, characterized in that... [Configuration 6] The type information is information representing whether the virtual object is the main subject. The image processing system according to any one of Configurations 1 to 4, characterized in that... [Configuration 7] The virtual object information changing unit erases the image of the virtual object based on the type information. The image processing system according to any one of Configurations 1 to 6, characterized in that... [Configuration 8] The virtual object information changing unit changes the image of the virtual object to an image stored in a storage device in advance based on the type information. The image processing system according to any one of Configurations 1 to 7, characterized in that... [Configuration 9] The virtual object information changing unit changes the image of the virtual object to an image selected from a plurality of images stored in a storage device in advance based on the type information. The image processing system according to any one of Configurations 1 to 8, characterized in that... [Configuration 10] It has a subject occlusion determination unit that determines whether the real object or the virtual object is occluded by another virtual object based on the position information of the virtual object. The virtual object information changing unit changes the transparency of part or all of the image of the virtual object that has not been erased based on the determination of the subject occlusion determination unit. The image processing system according to Configuration 7, characterized in that... [Configuration 11] The subject occlusion determination unit determines whether an object with a lower recording priority occludes an object with a higher recording priority based on the recording priorities and position information of the real object and the virtual object. The image processing system according to Configuration 10, characterized in that... [Configuration 12] The recording priority is information included in the type information, acquired from the outside, or generated by the type information generation unit. The image processing system according to Configuration 11, characterized in that... [Configuration 13] The recording priority is set such that the main subject is at the highest level. The image processing system according to Configuration 11 or 12, characterized in that... [Configuration 14] The subject occlusion determination unit evaluates the degree of occlusion in multiple stages. The virtual object information change unit changes the transparency of the virtual object image to a value corresponding to the degree of occlusion. The image processing system according to any one of Configurations 10 to 13, characterized in that... [Configuration 15] Having an automatic shooting control unit that automatically frames and records images. The image processing system according to any one of Configurations 1 to 14, characterized in that... [Configuration 16] The automatic shooting control unit controls the shooting angle and recording timing, excluding the image information of the virtual object whose image is erased by the virtual object information change unit. The image processing system according to Configuration 15, characterized in that... [Configuration 17] An image processing method executed by an image processing system, comprising: An imaging step of capturing a real object to generate a real image; A virtual object information acquisition step of acquiring display information of a virtual object from the outside; A virtual object information change step of changing the display information of the virtual object based on the type information of the virtual object; A virtual object synthesis step of synthesizing the real image and the virtual object image generated by the virtual object information change unit; A recording step of recording the synthesized image synthesized by the virtual object synthesis unit. An image processing method, characterized by including the above steps. [Configuration 18] A program for causing a computer to execute each step of the image processing method according to Configuration 17.

Explanation of Signs

[0118] 300: Image processing apparatus 114: Virtual object information changing unit 115: Virtual object synthesizing unit 116: Recording unit

Claims

1. An imaging unit that captures a real object to generate a real image, A virtual object information acquisition unit that acquires display information of a virtual object from the outside, A virtual object information changing unit that changes the display information of the virtual object based on the type information of the virtual object, A virtual object synthesizing unit that synthesizes the real image and the virtual object image generated by the virtual object information changing unit, A recording unit that records the composite image synthesized by the virtual object synthesizing unit, An image processing system having the above.

2. The type information of the virtual object is acquired by the virtual object information acquisition unit from the outside, The image processing system according to claim 1, characterized in that.

3. Further comprising a type information generation unit that generates type information of the virtual object based on the display information, The image processing system according to claim 1, characterized in that.

4. A plurality of the type information are set for the virtual object, The virtual object information changing unit changes the display information of the virtual object according to each of the plurality of type information, The virtual object synthesizing unit generates a plurality of composite images obtained by synthesizing the real image and each of the plurality of virtual object images generated by the virtual object information changing unit, The recording unit records each of the plurality of composite images, The image processing system according to claim 1, characterized in that.

5. The type information is information representing the security level of the virtual object, The image processing system according to claim 1, characterized in that.

6. The type information is information indicating whether the virtual object is the main subject, The image processing system according to claim 1, characterized in that.

7. The virtual object information changing unit erases the image of the virtual object based on the type information, The image processing system according to claim 1, characterized in that.

8. The virtual object information changing unit changes the image of the virtual object to an image stored in a storage device in advance based on the type information, The image processing system according to claim 1, characterized in that.

9. The virtual object information changing unit changes the image of the virtual object to an image selected from a plurality of images stored in a storage device in advance based on the type information, The image processing system according to claim 1, characterized in that.

10. Having a subject occlusion determination unit that determines whether the real object or the virtual object is occluded by another virtual object based on the position information of the virtual object, Based on the determination of the subject occlusion determination unit, the virtual object information changing unit changes the transparency of part or all of the image of the virtual object that has not been erased. The image processing system according to claim 7, characterized in that.

11. Based on the recording priorities and position information of the real object and the virtual object, the subject occlusion determination unit determines whether an object with a lower recording priority occludes an object with a higher recording priority. The image processing system according to claim 10, characterized in that.

12. The recording priority is information included in the type information, acquired from the outside, or generated by the type information generation unit. The image processing system according to claim 11, characterized in that.

13. For the recording priority, the main subject is set to the highest level. The image processing system according to claim 11, characterized in that.

14. The subject occlusion determination unit evaluates the degree of occlusion in multiple stages. Based on the degree of occlusion, the virtual object information changing unit changes the transparency of the virtual object image to a value corresponding to the degree of occlusion. The image processing system according to claim 10, characterized in that.

15. An automatic shooting control unit that automatically frames and records an image. The image processing system according to any one of claims 1 to 14, characterized in that.

16. Excluding the image information of the virtual object whose image is erased by the virtual object information changing unit, the automatic shooting control unit controls the angle of view and the recording timing. The image processing system according to claim 15, characterized in that.

17. An image processing method executed by an image processing system, comprising: An imaging step of capturing a real object to generate a real image; A virtual object information acquisition step of acquiring display information of a virtual object from the outside; A virtual object information change step of changing the display information of the virtual object based on the type information of the virtual object; A virtual object synthesis step of synthesizing the real image and the virtual object image generated by the virtual object information changing unit; A recording step of recording the composite image synthesized by the virtual object synthesis unit. An image processing method, characterized by including the above steps.

18. A program for causing a computer to execute each step of the image processing method according to claim 17.

Images