IMAGE PROCESSING SYSTEM, IMAGE PROCESSING METHOD, AND PROGRAM
The image processing system addresses the issue of incomplete subject extraction and coloring in virtual viewpoint images by generating foreground silhouette images and using device status-based transmission, ensuring high-quality virtual viewpoint videos.
Patent Information
- Application Number
- JP2020211474
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-12-21
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-12-21
AI Technical Summary
Existing image processing systems for generating virtual viewpoint images risk losing valuable data and deteriorating image quality due to incorrect preprocessing, particularly in subject extraction, leading to incomplete coloring of subjects in the generated 3D models.
An image processing system that generates foreground silhouette images and foreground images based on the status of the photographing device, including pixel information, to ensure accurate transmission and coloring of virtual viewpoint videos, even when preprocessing fails.
The system effectively suppresses deterioration in foreground image quality by ensuring complete subject extraction and coloring, even in cases of preprocessing errors, maintaining high video quality in virtual viewpoint videos.
Smart Images

Figure 0007581040000001 
Figure 0007581040000002 
Figure 0007581040000003
Abstract
Description
[Technical field]
[0001] The present invention relates to image processing system The present invention relates to an image processing method and a program. [Background technology]
[0002] Conventionally, to generate a virtual viewpoint video, multiple cameras (hereinafter referred to as physical cameras to distinguish them from virtual cameras) are placed around a subject, and images are captured from each physical camera according to the angle of view. Next, a virtual viewpoint video from an arbitrary virtual camera viewpoint is generated by image processing using the positional relationship and captured images of each physical camera.
[0003] Since the virtual viewpoint image is an image from the viewpoint of a virtual camera that is not dependent on the installation position of an actual physical camera, the user can freely select the viewpoint of the virtual camera, making it possible to generate images from free viewpoints that are not available in conventional images.
[0004] In Patent Document 1, a subject is extracted from an image captured by a physical camera, and the image of the extracted part of the subject is transmitted to a computer to generate a 3D model of the subject. At this time, the compression rate of the background part other than the subject is increased to reduce the amount of transmitted data. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2017-211827 A Summary of the Invention [Problem to be solved by the invention]
[0006] Patent Document 1 makes it possible to save computer resources such as transmission bandwidth and storage capacity compared to transmitting images captured by a physical camera as is. However, as mentioned above, when transmitting the results of some preprocessing such as subject extraction processing, if the preprocessing is not performed correctly, the amount of transmitted data can be reduced, but useful data may be lost.
[0007] For example, in the case of a process for extracting an object, if the detection of the object fails for some reason, the image information of the object from the physical camera that failed to detect the object will be lost. In this case, when a virtual viewpoint image is generated by performing a coloring process on a 3D model of the object, it becomes impossible to color the object using the image information that would have been obtained from the physical camera that failed to detect the object. As a result, it becomes impossible to color the object that exists only within the angle of view of the physical camera that failed to detect the object.
[0008] An object of the present invention is to make it possible to suppress deterioration in quality of a foreground image. [Means for solving the problem]
[0009] The image processing system of the present invention includes a first generation means for generating a foreground silhouette image showing a foreground area extracted from a photographed image photographed by a photographing device, the foreground silhouette image being used to generate a three-dimensional model of the foreground, and a transmission means for transmitting a foreground image corresponding to the foreground area in the photographed image, the foreground image being used to generate color information of the foreground included in a virtual viewpoint video, or the photographed image, based on a state of the photographing device; The status of the photographing device is the status of the number of pixels included in the foreground area in the photographed image. . Effect of the Invention
[0010] According to the present invention, deterioration in the quality of the foreground image can be suppressed. [Brief description of the drawings]
[0011] [Figure 1]1 is a block diagram showing an example of a functional configuration of an image processing device; [Diagram 2] FIG. 2 is an explanatory diagram showing the flow of an image processing method. [Diagram 3] 11A and 11B are diagrams illustrating a process performed when a defect occurs in pre-processing of a foreground image; [Figure 4] 11A and 11B are diagrams illustrating a process performed when a defect occurs in pre-processing of a foreground image; [Diagram 5] 13 is a flowchart of a data acquisition process. [Figure 6] FIG. 2 is an explanatory diagram showing the positional relationship of physical cameras. [Figure 7] FIG. 2 is an explanatory diagram showing the positional relationship of physical cameras. [Figure 8] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the image processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] (First embodiment) Fig. 1 is a block diagram showing an example of the functional configuration of an image processing device 100 according to the first embodiment, and Fig. 2 is a diagram showing the flow of an image processing method of the image processing device 100. The image processing device 100 has an imaging unit 101, a foreground extraction unit 102, a transmission unit 103, a foreground model generation unit 104, a storage unit 105, an output video generation unit 106, a shooting state determination unit 107, and a foreground image control unit 108.
[0013] The imaging unit 101 has a plurality of physical cameras, which are arranged to surround a subject and synchronously capture the subject from a plurality of directions. However, the number and arrangement of the physical cameras are not limited. The imaging unit 101 generates a plurality of captured images 201 by capturing images with the plurality of physical cameras and outputs the generated captured images 201 to the foreground extraction unit 102.
[0014] Foreground extraction unit 102 extracts a subject as a "foreground" from multiple captured images 201 generated by multiple physical cameras of imaging unit 101, and generates multiple foreground masks 202 indicating areas where the foreground exists. Then, foreground extraction unit 102 outputs the multiple foreground masks 202 to transmission unit 103. Transmission unit 103 generates multiple foreground images 204 using the multiple foreground masks 202 for the multiple captured images 201. Foreground images 204 are images in which a subject is extracted as a "foreground" from captured image 201.
[0015] In the case of soccer as an example, a person and a ball are extracted as the foreground in foreground image 204. The content of extracted foreground image 204 is the same as the foreground content in photographed image 201. Foreground mask 202 is an image made up of binary data in which the pixel values of the person and ball areas in foreground image 204 of photographed image 201 are "1" as a mask area, and the pixel values of areas other than foreground image 204 are "0". This foreground mask 202 is used as a silhouette image when foreground model generation unit 104 performs model generation processing using VisualHull or the like.
[0016] Note that foreground image 204 and foreground mask 202 are not limited in format as image data. For example, foreground image 204 and foreground mask 202 are not limited in bit depth per pixel (8 bits, 10 bits, etc.), format expressing color per pixel (YUV444, YUV422, RGB, etc.), color of area other than the foreground, or color of mask part. Also, image file format of foreground image 204 and foreground mask 202 is general PNG (Portable Network Graphics), JPEG (Joint Photographic Experts Group), etc. The above image file format may be other unique formats, etc., and is not limited.
[0017] The foreground extraction unit 102 may extract the foreground by using background difference information. Most simply, the foreground extraction unit 102 stores in advance a background image captured by the imaging unit 101 in a state where the foreground does not exist, and calculates the difference between the captured image 201 in which the foreground exists and the background image. The foreground extraction unit 102 then determines that the position of a pixel where the difference is greater than a threshold value is the foreground, and generates a foreground mask 202. The foreground may be extracted by using other methods such as a feature amount on an image related to a subject or machine learning, and the method of extracting the foreground is not limited.
[0018] Transmission unit 103 generates a plurality of foreground images 204 using a plurality of foreground masks 202 for a plurality of captured images 201, and compresses the plurality of foreground images 204. Then, transmission unit 103 transmits the compressed plurality of foreground images 204 and the plurality of foreground masks 202 to foreground model generation unit 104. For example, transmission unit 103 is connected to foreground model generation unit 104 via a network. The network connection is, for example, Ethernet (registered trademark), which is most commonly used in computer networks, but is not limited to this.
[0019] The foreground model generating unit 104 generates a three-dimensional model 203 by VisualHull using the multiple foreground masks 202 transmitted by the transmitting unit 103. The processing of VisualHull is as described above.
[0020] Storage unit 105 stores 3D model 203 generated by foreground model generation unit 104 and a plurality of compressed foreground images 204 transmitted by transmission unit 103. Storage unit 105 stores 3D model 203 and foreground images 204 in a storage device. Specifically, storage unit 105 stores 3D model 203 and foreground images 204 in a Hard Disk Drive (HDD), a Solid State Drive (SSD), a flash memory, or the like. The storage device does not have to be a single device, and may be a collection of a plurality of physical devices. In addition, the storage device may be a mixture of a plurality of types of devices, or may be a device on a cloud virtualized on a network.
[0021] The output video generating unit 106 expands the compressed foreground images 204 stored by the storage unit 105. The output video generating unit 106 then uses color information of the expanded foreground images 204 to color the 3D model 203 of the subject stored by the storage unit 105, generating a virtual viewpoint video 205. For example, the output video generating unit 106 generates a distance image representing the distance between each physical camera and each point constituting the 3D model 203. The output video generating unit 106 then selects a physical camera that is closest to each point of the 3D model 203 from the distance image, obtains color information of a corresponding pixel position from the foreground image 204 of the physical camera, and colors the 3D model 203. As for the method of coloring, a method of blending the colors of the captured images 201 of the multiple physical cameras that are close to each other may be used, but the method of coloring the 3D model 203 is not limited. The flow of generating the virtual viewpoint video 205 is as described above.
[0022] In this embodiment, shooting state determination unit 107 determines the shooting state in which shooting is being performed by imaging unit 101, and outputs the determination result to foreground image control unit 108. Foreground image control unit 108 controls the processing performed by transmission unit 103 according to the shooting state determination result by shooting state determination unit 107. Transmission unit 103 generates foreground image 204 using foreground mask 202 or foreground image 204 without using foreground mask 202 according to the control of foreground image control unit 108.
[0023] Next, a description will be given with reference to FIG. 2. The captured image 201 is an image captured by a plurality of physical cameras in the imaging unit 101. The foreground extraction unit 102 generates a plurality of foreground masks 202 based on the plurality of captured images 201. The transmission unit 103 generates a plurality of foreground images 204 using the plurality of foreground masks 202 for the plurality of captured images 201. The foreground images 204 and the foreground masks 202 are generated in the same number as the number of physical cameras in the imaging unit 101. The foreground model generation unit 104 generates a three-dimensional model 203 based on the plurality of foreground masks 202. The three-dimensional models 203 are generated in the same number as the number of subjects. In this case, since the scene has two people, two three-dimensional models 203 are generated.
[0024] Foreground image 204 is compressed by transmission unit 103. Specifically, transmission unit 103 distinguishes between an area where the foreground exists and other areas for photographed image 201 by using foreground mask 202. Here, the area where the foreground exists is defined as the foreground area. Transmission unit 103 generates foreground image 204 by converting areas other than the foreground area for photographed image 201 into a single color (e.g., black). Transmission unit 103 enhances compression efficiency by compressing foreground image 204. For compressing foreground image 204, a lossless compression method is preferable to prevent loss of image information, but a lossy compression method may also be used. Examples of lossless compression methods include run-length and Huffman coding. Also, examples of lossy compression methods include a method of performing a Fourier transform to convert to a frequency domain representation and then deleting areas that cannot be perceived by humans. Specifically, JPEG (Joint Photographic Experts Group) is used as a lossy compression method for still images. In addition to those described above, there are various other compression methods, both lossless and lossy, and the compression method is not limited thereto.
[0025] Transmitter 103 can reduce the amount of transmission data by compressing foreground image 204. Virtual viewpoint video 205 is generated by output video generator 106 as a result of coloring 3D model 203 using foreground image 204 in accordance with the virtual viewpoint specified by the user.
[0026] FIG. 3 is a diagram showing how image quality degradation occurs in the virtual viewpoint image when the foreground extraction process, which is a pre-processing before transmission, fails. A plurality of captured images 301 are images captured by a plurality of physical cameras of the imaging unit 101. The foreground extraction unit 102 may fail to extract the foreground from the captured image 301 and generate a foreground mask 302 of a certain physical camera. In the foreground mask 302, the head part 306 of the person on the right side is black, indicating that no subject exists in the head part 306 of the person on the right side. The transmission unit 103 generates a foreground image 304 for the captured image 301 using the foreground mask 302. When the head part 306 of the foreground mask 302 is black, the head part 307 of the person on the right side of the foreground image 304 is treated as a part where no foreground exists. As a result, the head portion 307 of the person on the right side of the foreground image 304 is converted to black, and the image information of the head portion 307 of the person on the right side that existed at the time of the photographed image 301 is lost.
[0027] For example, output video generation unit 106 may generate virtual viewpoint video 305 from a viewpoint close to that of the physical camera of foreground image 304 from which foreground extraction has failed. In this case, output video generation unit 106 cannot color three-dimensional model 303 using head part 307 of the subject on the right side of foreground image 304, so head part 308 of the person on the right side in generated virtual viewpoint video 305 is not colored.
[0028] Depending on the coloring function of the output video generating unit 106, for example, when the 3D model 303 cannot be colored, it is possible to use the color of the foreground image 304 of another physical camera. That is, the output video generating unit 106 can suppress failure in coloring the virtual viewpoint video 305, which is the final product, by coloring using another foreground image 304 that includes the angle of view at the desired virtual viewpoint. However, when there is a range of the angle of view captured only by the foreground image 304 corresponding to the physical camera that failed to extract the foreground, it may be impossible to interpolate using the color information of the foreground image 304 corresponding to the other physical camera. For example, when the range of the angle of view captured only by the foreground image 304 of a certain physical camera includes the head part 306 on the right side of the subject, it is impossible to use the color information of the captured image 401 of the other physical camera.
[0029] 4 is a diagram showing how to handle a case where foreground extraction fails using shooting state determination unit 107 and foreground image control unit 108 according to this embodiment. A plurality of captured images 401 are images captured by a plurality of physical cameras of imaging unit 101. Foreground extraction unit 102 generates foreground mask 402 based on the difference between captured image 401 and a background image captured in advance. In foreground mask 402, pixel values of areas in captured image 401 where the foreground exists are 1, and pixel values of areas where the foreground does not exist are 0.
[0030] The shooting state determination unit 107 determines the shooting state of the captured image 401 based on the foreground mask 402 generated by the foreground extraction unit 102. Specifically, the shooting state determination unit 107 determines the shooting state based on the number of pixels in the area where the foreground exists in the foreground mask 402. If the number of pixels in the area where the foreground exists is less than a predetermined foreground presence threshold, it is considered that the foreground does not exist in the captured image 401. If the number of pixels in the area where the foreground exists is more than the foreground presence threshold but less than the predetermined foreground amount determination threshold, it indicates that the foreground area is small relative to the entire angle of view of the captured image 401.
[0031] When the foreground area is small relative to the overall angle of view of the captured image 401, it indicates that the foreground subject cannot be expressed with high resolution, and therefore the shooting difficulty is considered to be high. Therefore, when the number of pixels in the area where the foreground exists is greater than the foreground presence threshold and less than the foreground amount determination threshold, the shooting state determination unit 107 determines that the shooting difficulty is high. Also, when the number of pixels in the area where the foreground exists is greater than the foreground amount determination threshold, the shooting state determination unit 107 determines that the shooting difficulty is low.
[0032] Conversely, there may be a case where a large amount of rain or snow that is not desired to be extracted as the foreground is extracted as the foreground due to the influence of weather, resulting in a very large number of pixels in the area where the foreground exists. In this case, if the number of pixels in the area where the foreground exists is greater than a predetermined foreground saturation determination threshold, the shooting state determination unit 107 determines that the shooting difficulty is high. Also, if the number of pixels in the area where the foreground exists is less than a predetermined foreground saturation determination threshold, the shooting state determination unit 107 determines that the shooting difficulty is low.
[0033] When the number of pixels in the area where the foreground exists is greater than the foreground amount judgment threshold and less than the foreground saturation judgment threshold, the shooting state judgment unit 107 judges that the shooting difficulty level is low. When the number of pixels in the area where the foreground exists is less than the foreground amount judgment threshold or greater than the foreground saturation judgment threshold, the shooting state judgment unit 107 judges that the shooting difficulty level is low.
[0034] In addition, assuming a case where a sports game is shot, the maximum number of pixels in the foreground area can be calculated in advance depending on the angle of view and lens performance of the installed physical camera. The number of pixels calculated in advance can be used as the foreground saturation determination threshold. The shooting difficulty differs depending on the lens and arrangement of each physical camera, so it is calculated for each physical camera. The shooting difficulty for each physical camera calculated in this way is output from the shooting state determination unit 107 to the foreground image control unit 108.
[0035] The method of calculating the degree of difficulty of taking pictures is not limited to the method of using the number of pixels in the area where the foreground exists, which is used by the shooting state determination unit 107. The shooting state determination unit 107 calculates the similarity between the distribution of color information of the area where the foreground exists in the photographed image 401 and the distribution of color information of the background image of the area where the foreground exists in the photographed image 401, and determines the degree of difficulty of taking pictures. If the similarity between the distributions of the color information of both is higher than a threshold value, the shooting state determination unit 107 determines that the shooting difficulty is high, since it indicates that the color of the foreground and the color of the background are similar. If the similarity between the distributions of the color information of both is lower than a threshold value, the shooting state determination unit 107 determines that the shooting difficulty is low.
[0036] When it is determined that the state is one in which the degree of difficulty of photographing is high, foreground image control unit 108 instructs transmission unit 103 to generate foreground image 404 in which the degree of difficulty of photographing is high. Also, when it is determined that the state is one in which the degree of difficulty of photographing is low, foreground image control unit 108 instructs transmission unit 103 to generate foreground image 404 in which the degree of difficulty of photographing is low.
[0037] When the difficulty level of shooting is high, transmitting unit 103 does not use foreground mask 402 for image 401 captured by a physical camera with a high shooting difficulty, generates captured image 401 itself as foreground image 404, and compresses foreground image 404. Also, when the difficulty level of shooting is low, transmitting unit 103 uses foreground mask 402 for image 401 captured by a physical camera with a low shooting difficulty, converts an area where no foreground exists into a single color (for example, black), and generates foreground image 404, and compresses foreground image 404.
[0038] When the shooting difficulty is high, it becomes impossible to improve the compression efficiency by making the foreground-free area of foreground image 404 monochromatic, but it is possible to suppress the loss of head part 407 of foreground image 404 corresponding to head part 406 of the person on the right side that is lost in foreground mask 402. Whether or not to generate foreground image 404 using foreground mask 402 is determined according to the shooting difficulty of each physical camera. Thereafter, transmission unit 103 performs compression processing such as the above-mentioned run-length compression on foreground images 404 of all physical cameras, and transmits the compressed multiple foreground images 404 and multiple foreground masks 402 to foreground model generation unit 104.
[0039] Foreground model generation unit 104 generates 3D model 403 based on the plurality of foreground masks 402 transmitted by transmission unit 103. Storage unit 105 stores 3D model 403 generated by foreground model generation unit 104 and the plurality of compressed foreground images 404 transmitted by transmission unit 103.
[0040] Output video generation unit 106 expands multiple compressed foreground images 404 stored by storage unit 105. Then, output video generation unit 106 generates virtual viewpoint video 405 by applying color to 3D model 403 of the subject stored by storage unit 105 using color information of multiple expanded foreground images 404 according to a virtual viewpoint specified by the user. In this case, output video generation unit 106 can also color head portion 407 of foreground image 404 corresponding to head portion 406 of the person on the right side missing in foreground mask 402, making it possible to suppress deterioration in quality of final virtual viewpoint video 405.
[0041] 5 is a flowchart showing an image processing method of the image processing device 100. First, in step S501, the imaging unit 101 generates a plurality of captured images 401 by imaging processing of a plurality of physical cameras. The foreground extraction unit 102 functions as a foreground mask generation unit, and generates a plurality of foreground masks 402 indicating an area in which the foreground exists for each of the plurality of captured images 401 captured by the plurality of physical cameras. Specifically, the foreground extraction unit 102 generates a plurality of foreground masks 402 for each of the plurality of physical cameras based on the difference between the plurality of captured images 401 generated by the imaging unit 101 and a plurality of background images captured in advance.
[0042] Next, in step S502, the shooting state determining unit 107 determines the shooting difficulty level of each of the physical cameras based on the number of pixels in the area where the foreground exists in the foreground masks 402.
[0043] Next, in step S503, the image processing device 100 starts a process of sequentially performing loop processes for the plurality of physical cameras, the number of which corresponds to the number of the physical cameras.
[0044] Next, in step S504, foreground image control unit 108 determines, for each physical camera, whether the shooting difficulty of captured image 401 of the target physical camera is lower than a threshold. If the shooting difficulty of captured image 401 is lower than the threshold, foreground image control unit 108 instructs transmission unit 103 to that effect, and proceeds to step S505. If the shooting difficulty of captured image 401 is higher than the threshold, foreground image control unit 108 instructs transmission unit 103 to that effect, and transmission unit 103 generates captured image 401 of the target physical camera itself as foreground image 404 of the target physical camera, and proceeds to step S506.
[0045] In step S505, the transmitting unit 103 generates an image of a region in the captured image 401 of the target physical camera where the foreground exists, as the foreground image 404 of the target physical camera, based on the foreground mask 402 of the target physical camera. Specifically, the transmitting unit 103 converts pixel values of a region in the captured image 401 of the target physical camera where the foreground does not exist in the foreground mask 402 of the target physical camera into a single color, thereby generating the foreground image 404 of the target physical camera, and proceeds to step S506.
[0046] In step S506, the transmitter 103 compresses (eg, run-length compresses) the foreground image 404 of the target physical camera, and returns to the loop process of step S503 to repeat the process for the next target physical camera.
[0047] In step S506, the transmitting unit 103 may perform compression processing of different types on the foreground image 404 of the target physical camera depending on the level of the shooting difficulty of the target physical camera. For example, when the shooting difficulty of the target physical camera is low, the transmitting unit 103 can improve the compression efficiency by performing run-length compression processing on the foreground image 404 of the target physical camera since the foreground image 404 includes a monochromatic area. In addition, when the shooting difficulty of the target physical camera is high, the transmitting unit 103 can suppress a decrease in compression efficiency by performing lossy compression processing on the foreground image 404 of the target physical camera since the foreground image 404 is the same as the captured image 401. The lossy compression processing is effective when the transmission band from the transmitting unit 103 to the foreground model generating unit 104 is insufficient or when it is desired to further save the capacity for the storage unit 105 to store the compressed foreground image 404.
[0048] According to this embodiment, the image processing device 100 is capable of suppressing degradation of image quality in the final virtual viewpoint image even in cases where the foreground extraction unit 102 fails to process the captured image 401 by the physical camera.
[0049] Second Embodiment In the second embodiment, an embodiment will be described in which the method of determining the shooting state in the shooting state determination unit 107 described in the first embodiment is different. The image processing device 100 according to the second embodiment has a similar configuration to the image processing device 100 according to the first embodiment, but the processing of the shooting state determination unit 107 shown in FIG. 1 is different.
[0050] FIG. 6 is a diagram showing a state where a soccer match is being photographed on a soccer field 601. The process in the photographing state determination unit 107 will be described with reference to FIG. 6. The imaging unit 101 has physical cameras 602, 603, and 604 that have similar angles of view 605, 606, and 607 among a plurality of physical cameras. Whether the angles of view 605 to 607 are similar or not is determined in advance by a user visually judging the contents of the images when the physical cameras 602 to 604 are installed. Whether the angles of view 605 to 607 are similar or not may be determined, as a method other than visual judgment, from information on the installation position and the range of the angle of view in the camera parameters as a calibration result for calibrating the installation position and the attitude of the physical cameras 602 to 604. In addition, whether the angles of view 605 to 607 are similar or not may be determined based on the number of feature points that correspond between the physical cameras 602 to 604 on the photographed image 401 photographed by the physical cameras 602 to 604. The method of determining whether the angles of view 605 to 607 are similar or not is not limited.
[0051] Physical camera 602 captures an image within a range of angle of view 605. Physical camera 603 captures an image within a range of angle of view 606. Physical camera 604 captures an image within a range of angle of view 607. At this time, the image capturing state determination unit 107 acquires information for determining the image capturing state from the physical cameras 602 to 604 present in the imaging unit 101. The information for determining the image capturing state is image capturing state information, and specifically has three values: normal, abnormal, and image capturing not possible.
[0052] 6, physical cameras 602 and 604 output "imaging impossible" as the imaging state information to imaging state determination unit 107. Physical camera 603 outputs "normal" as the imaging state information to imaging state determination unit 107. At this time, there is no other physical camera with a similar angle of view that can cover angle of view 606 of physical camera 603, and it is considered that this is a situation in which loss of image information of physical camera 603 should be avoided. At this time, imaging state determination unit 107 determines that the imaging difficulty level of physical camera 603 is high, and outputs the imaging difficulty level to foreground image control unit 108.
[0053] The photographing state determination unit 107 determines the level of difficulty of photographing according to information on whether the captured images 401 of the physical cameras 602-604 having similar angles of view 605-607 among the plurality of physical cameras are normal, abnormal, or impossible to photograph. For example, when the captured images 401 of all the physical cameras 602-604 are normal, the photographing state determination unit 107 determines that the state of difficulty of photographing is low. On the other hand, when the captured images 401 of some of the physical cameras 602-604 are abnormal or impossible to photograph, the photographing state determination unit 107 determines that the state of difficulty of photographing is high.
[0054] As in the first embodiment, when it is determined that the photographing difficulty level is high, foreground image control unit 108 instructs transmission unit 103 to perform processing for when the photographing difficulty level is high. The processing of transmission unit 103 when the photographing difficulty level is high is the same as that described in the first embodiment.
[0055] The shooting state information output by the imaging unit 101 to the shooting state determination unit 107 may be the shot image 401 itself. In this case, the shooting state determination unit 107 analyzes the shot image 401 and determines whether or not the shot image 401 is being shot normally. For example, the shooting state determination unit 107 compares the shot image 401 of the current frame with the shot image 401 of the previous frame in the time direction, and determines that the shooting state information is abnormal if there is a sudden change in the content of the shot image 401 that is not expected due to the movement of the subject. Specifically, the shooting state determination unit 107 determines that the shooting state information is abnormal if the front of the physical camera is covered by a bird or a cheering flag, or if an abnormal change in color or noise occurs due to an abnormality in the imaging sensor of the physical camera. In addition, the shooting state determination unit 107 can determine that shooting is not possible as the shooting state information if the shot image 401 cannot be obtained from the physical camera in the first place. As described above, the photographing state determining unit 107 can determine whether the photographed image 401 is normal, abnormal, or unphotographable based on the photographed image 401 .
[0056] The shooting state determination unit 107 may determine the similarity relationships between the angles of view 605-607 of the physical cameras 602-604 of the imaging unit 101 when the physical cameras 602-604 are installed and store the similarity relationships in advance in a format such as a table. If the physical cameras 602-604 of the imaging unit 101 change the angles of view during shooting, it is necessary to re-determine the similarity relationships between the angles of view 605-607 every time the angles of view change.
[0057] The user may specify the shooting difficulty level for each physical camera. When the user specifies the shooting difficulty level, the user may specify only the shooting difficulty level for an arbitrary number of physical cameras, instead of specifying the shooting difficulty level for all physical cameras, and the shooting state determination unit 107 may determine the shooting difficulty level for the remaining physical cameras by the above-mentioned method. When the image processing device 100 cannot detect the shooting difficulty level, but the user wants to specify the shooting difficulty level for some reason that the user recognizes, a method in which the user specifies the shooting difficulty level is effective.
[0058] According to this embodiment, the image processing device 100 is capable of suppressing the degradation of image quality that occurs in the final virtual viewpoint image when the impact is significant when loss of image information occurs due to the shooting conditions of the physical camera.
[0059] (Third embodiment) The third embodiment is different from the first embodiment in the method of determining the shooting state by the shooting state determination unit 107. The image processing device 100 according to the third embodiment has a similar configuration to the image processing device 100 according to the first embodiment, but differs in the processing of the shooting state determination unit 107 in FIG.
[0060] FIG. 7 is a diagram showing a state in which a soccer match is being shot on a soccer field 701. The process of the shooting state determination unit 107 will be described with reference to FIG. 7. A physical camera with a viewing angle 702 and a physical camera with a viewing angle 703 are installed on the soccer field 701. At this time, the viewing angle 702 is a viewing angle that can capture an image in front of the goal. The viewing angle 703 is a viewing angle at the center line position and a portion closer to the sideline. Among images during a match, images that are particularly highly watched by viewers and considered important are images of goal scenes, etc., and images with a viewing angle in front of the goal are considered to be important. Here, the shooting importance is set to high or low according to the importance of the viewing angle shot by the physical camera. This shooting importance is output from each physical camera of the imaging unit 101 to the shooting state determination unit 107. The shooting importance of the physical camera with the viewing angle 702 is high. The shooting importance of the physical camera with the viewing angle 703 is low.
[0061] The shooting state determination unit 107 outputs the shooting importance of the captured image 401 of each physical camera to the foreground image control unit 108. For the captured image 401 of the physical camera with a high shooting importance, the foreground image control unit 108 instructs the transmission unit 103 to convert the captured image 401 itself into the foreground image 404. For the captured image 401 of the physical camera with a low shooting importance, the foreground image control unit 108 instructs the transmission unit 103 to generate the foreground image 404 by converting an area where the foreground does not exist into a single color. The processing of the transmission unit 103 is the same as that of the first embodiment. The image processing device 100 can suppress a situation in which video information is lost for the captured image 401 of the physical camera with a high shooting importance, regardless of the result of the processing of the foreground extraction unit 102.
[0062] Furthermore, whenever the angle of view captured by the physical cameras of the imaging unit 101 changes, the shooting state determination unit 107 determines the shooting importance of the captured image 401 each time, based on the captured image 401. Specifically, the shooting state determination unit 107 defines in advance areas of the soccer field 701 that have a high shooting importance, and when the captured image 401 of each physical camera includes an area with a high shooting importance, the shooting state determination unit 107 determines that the captured image 401 has a high shooting importance.
[0063] According to this embodiment, the image processing device 100 generates a foreground image 404 taking into account the importance of capturing a subject captured by a physical camera, thereby making it possible to suppress degradation of image quality in the final virtual viewpoint image.
[0064] In addition, in any of the above-described embodiments, the resolution of foreground image 404 is maintained as it is in photographed image 401, but foreground image 404 may be divided into regions. For example, a case will be described in which only a rectangular portion circumscribing pixels in photographed image 401 where the foreground exists is treated as the foreground image and the region is divided. In this case, when foreground image 404 is divided into two rectangular images, image processing device 100 may perform the processing in the above-described embodiment on each of the rectangular units of the foreground image.
[0065] In this way, by handling foreground image 404 as divided regions, more detailed handling becomes possible. Specifically, if foreground image 404 is handled in the resolution unit of a physical camera, image processing device 100 controls the processing of foreground image 404 in physical camera units. In contrast, if foreground image 404 is handled by dividing regions, image processing device 100 can control the generation of foreground image 404 in rectangular units that are finer than physical camera units.
[0066] FIG. 8 is a block diagram showing an example of the hardware configuration of a computer applicable to foreground extraction unit 102, transmission unit 103, foreground model generation unit 104, output video generation unit 106, shooting state determination unit 107, and foreground image control unit 108 according to each of the above embodiments.
[0067] CPU 801 controls the entire computer using computer programs and data stored in RAM 802 and ROM 803. In addition, CPU 801 executes the computer programs to execute the processes performed by foreground extraction unit 102, transmission unit 103, foreground model generation unit 104, output video generation unit 106, shooting state determination unit 107, and foreground image control unit 108 according to the above embodiments.
[0068] The RAM 802 has an area for temporarily storing computer programs and data loaded from an external storage device 804, or data acquired from the outside via an I / F (interface) 805. The RAM 802 also has a work area used when the CPU 801 executes various processes. That is, the RAM 802 can be allocated as a frame memory, for example, or can provide various other areas as appropriate. The ROM 803 stores computer setting data, a boot program, and the like.
[0069] The external storage device 804 is a large-capacity information storage device, such as a hard disk drive. The external storage device 804 stores an operating system (OS) and computer programs for causing the CPU 801 to realize the functions of each unit shown in Fig. 1. Furthermore, the external storage device 804 may store image data to be processed.
[0070] Computer programs and data stored in the external storage device 804 are loaded into the RAM 802 as appropriate under the control of the CPU 801 and are processed by the CPU 801 .
[0071] The I / F 805 can be connected to a network such as a LAN or the Internet, and other devices such as a projector or display device. The computer can obtain and transmit various information via the I / F 805. The bus 806 is a bus that connects the above-mentioned units. The storage unit 105 in FIG. 1 may be an external storage device 804, or a storage device via an external network via the I / F 805. The operation of the above-mentioned configuration is controlled mainly by the CPU 801, as described in the above embodiment.
[0072] Note that the first computer may realize foreground extraction unit 102, transmission unit 103, shooting state determination unit 107, and foreground image control unit 108, and the second computer may realize foreground model generation unit 104 and output video generation unit 106. In that case, the first computer and the second computer each have the configuration shown in FIG. 8 and are network-connected via I / F 805.
[0073] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.
[0074] It should be noted that the above-mentioned embodiments are merely illustrative of specific examples of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be embodied in various forms without departing from its technical concept or main features. [Explanation of symbols]
[0075] 101 imaging unit, 102 foreground extraction unit, 103 transmission unit, 104 foreground model generation unit, 105 storage unit, 106 output image generation unit, 107 shooting state determination unit, 108 foreground image control unit
Claims
1. A first generation means for generating a foreground silhouette image showing a foreground area extracted from a photographed image photographed by a photographing device, the foreground silhouette image being used to generate a three-dimensional model of the foreground; a transmission means for transmitting a foreground image corresponding to the foreground area in the captured image and used to generate color information of the foreground included in a virtual viewpoint video, or the captured image, based on a state of the image capturing device; 11. An image processing system according to claim 10, wherein the status of the photographing device is a status of the number of pixels included in the foreground area in the photographed image.
2. 2. The image processing system according to claim 1, wherein the transmission means transmits the foreground image when the number of pixels is greater than a first threshold, and transmits the captured image when the number of pixels is less than the first threshold.
3. 3. The image processing system according to claim 1, wherein the transmission means transmits the foreground image when the number of pixels is less than a second threshold, and transmits the captured image when the number of pixels is more than the second threshold.
4. 4. The image processing system according to claim 1, wherein the transmission means transmits the foreground image when the number of pixels is within a predetermined range, and transmits the captured image when the number of pixels is outside the predetermined range.
5. A first generation means for generating a foreground silhouette image showing a foreground area extracted from a photographed image photographed by a photographing device, the foreground silhouette image being used to generate a three-dimensional model of the foreground; a transmission means for transmitting a foreground image corresponding to the foreground area in the captured image and used to generate color information of the foreground included in a virtual viewpoint video, or the captured image, based on a state of the image capturing device; An image processing system characterized in that the status of the photographing device is a status of similarity between a distribution of color information of the foreground area of the photographed image and a distribution of color information of a background area of the photographed image.
6. 6. The image processing system according to claim 5, wherein the transmission means transmits the foreground image when the similarity is lower than a third threshold, and transmits the captured image when the similarity is higher than the third threshold.
7. A first generation means for generating a foreground silhouette image showing a foreground area extracted from a photographed image photographed by a photographing device, the foreground silhouette image being used to generate a three-dimensional model of the foreground; a transmission means for transmitting a foreground image corresponding to the foreground area in the captured image and used to generate color information of the foreground included in a virtual viewpoint video, or the captured image, based on a state of the image capturing device; An image processing system, wherein the situation of the image capturing device is a capturing situation of an image capturing device having an angle of view similar to that of the image capturing device.
8. The image processing system according to claim 7, characterized in that the transmission means transmits the foreground image when all of the photographing devices having a similar angle of view to the photographing device are normal, and transmits the photographed image when some of the photographing devices having a similar angle of view to the photographing device are abnormal or unable to photograph.
9. A first generation means for generating a foreground silhouette image showing a foreground area extracted from an image captured by a photographing device, the foreground silhouette image being used to generate a three-dimensional model of the foreground; a transmission means for transmitting a foreground image corresponding to the foreground area in the captured image and used to generate color information of the foreground included in a virtual viewpoint video, or the captured image, based on a state of the image capturing device; the status of the photographing device is a status of a photographing area of the photographing device, The image processing system according to claim 1, wherein the transmission means transmits the foreground image when the importance of the photographed area is high, and transmits the photographed image when the importance of the photographed area is low.
10. a model generating means for generating the three-dimensional model based on the foreground silhouette image; The image processing system according to any one of claims 1 to 9, further comprising an image generation means for generating the virtual viewpoint image, the image generation means generating color information of the foreground contained in the virtual viewpoint image based on the three-dimensional model and the transmitted foreground image or the transmitted captured image.
11. A first generation means for generating a foreground silhouette image showing a foreground area extracted from an image captured by a photographing device, the foreground silhouette image being used to generate a three-dimensional model of the foreground; a transmission means for transmitting a foreground image corresponding to the foreground area in the captured image and used to generate color information of the foreground included in a virtual viewpoint video, or the captured image, based on a state of the image capturing device; The image processing system is characterized in that, when transmitting the foreground image, the transmission means compresses the foreground image using a lossless compression process and transmits it, and, when transmitting the photographed image, compresses the photographed image using a lossy compression process and transmits it.
12. a first generation step of generating a foreground silhouette image showing a foreground area extracted from an image captured by an image capturing device, the foreground silhouette image being used to generate a three-dimensional model of the foreground; a transmission step of transmitting a foreground image corresponding to the foreground area of the captured image and used for generating color information of the foreground included in a virtual viewpoint video, or the captured image, based on a state of the image capturing device; 11. An image processing method for an image processing system, wherein the status of the photographing device is a status of the number of pixels included in the foreground area in the photographed image.
13. A program for causing a computer to function as each of the means of the image processing system according to any one of claims 1 to 11.
Citation Information
Patent Citations
Information processing unit, control method and program
JP2017211827A
Material generation apparatus, image generation apparatus and image processing apparatus
JP2020134973A
