Image processing system and image processing method

Abstract

To generate silhouette images with which highly accurate three-dimensional shape data can be generated while suppressing implementation cost of image processing apparatuses.SOLUTION: An image processing system 100 includes: a first image processing apparatus configured to generate data of a first silhouette image representing a region in which an object exists in a first input image by inputting data of an image, as data of the first input image, into a trained model, the image being obtained by first imaging means imaging the object, the first imaging means imaging a region including at least part of a specific region; and a second image processing apparatus configured to output data of a second silhouette image representing a region in which the object exists in a second input image by calculating a difference between the second input image and a background image captured by second imaging means in a state in which the object does not exist, using data of an image, as data of the second input image, which is obtained by the second imaging means that images the object, the second imaging means being different from the first imaging means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a technique for generating a silhouette image indicating a foreground region in an image.

Background Art

[0002] There is a technique for generating 3D shape data indicating the 3D shape of an object of interest (hereinafter simply referred to as an "object") using data of a plurality of images (hereinafter referred to as "imaging images") obtained by synchronized imaging by an imaging device. The 3D shape data is generated by a method such as a volume intersection method using a silhouette image generated by extracting a region corresponding to the object (hereinafter referred to as an "object region") from each imaging image. As a method for generating a silhouette image from an imaging image, for example, there are a background subtraction method or a machine learning method. In the background subtraction method, in the imaging angle imaged by a certain imaging device, an imaging image obtained by imaging during a period when no object exists is used as a background image, and a difference is taken between this background image and an imaging image obtained by imaging during a period when an object exists. Further, based on this difference, the object region and the background region in the imaging image are separated to generate a silhouette image indicating the object region (hereinafter referred to as a "silhouette image of the object"). In the machine learning method, first, a sufficient number of learning data pairs are prepared, each pair consisting of an imaging image obtained by imaging during a period when an object exists and data indicating the object region in the imaging image, which is teacher data. Subsequently, a learned model, which is the result of training a learning model using the learning data, is generated. Further, using this learned model, the object region and the background region in the imaging image are separated to generate a silhouette image of the object.

[0003] Depending on the characteristics included in the field of view of the imaging device, the most suitable separation method for separating the object region and the background region in the captured image will differ. When using background subtraction as the separation method, the object region may not be detected in the captured image in the following areas (hereinafter referred to as "undetected object"). For example, areas where there are objects with little movement, areas where there is a fixed background with a color similar to the object's color, and areas where there is movement and part of the background has a color similar to the object's color. In addition, in the following areas, areas other than the object region may be mistakenly detected as the foreground region in the captured image (hereinafter referred to as "false detection of objects"). For example, areas where the shadow of an object is generated, and areas where a virtual image is generated due to the reflection of the object's image from a glossy floor or a wet field surface.

[0004] Failure to detect or falsely detect these objects can lead to missing or incomplete 3D shape data. Therefore, it is useful to perform separation processing using machine learning methods in areas where failure or false detection may occur in separation processing using background subtraction. On the other hand, separation processing using machine learning generally has a higher computational cost than separation processing using background subtraction. In addition, because separation processing using machine learning infers the shape of an object from statistical information of surrounding pixels, the accuracy of the object region boundary in the silhouette image deteriorates compared to separation processing using background subtraction. Patent Document 1 discloses a technique for dividing the area to be separated by background subtraction and the area to be separated by machine learning within the field of view of a single imaging device. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2021-56960 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, the technology disclosed in Patent Document 1 requires that an image processing device corresponding to a single imaging device be equipped with a configuration for performing both background subtraction separation and machine learning separation, which increases the implementation cost for a single image processing device. [Means for solving the problem]

[0007] The image processing system related to this disclosure is Image the area of ​​the baseball field, including the clay surface. A first image processing apparatus comprising: an acquisition means for acquiring a first input image obtained by imaging an object with a first imaging means; and a first generation means for generating a first silhouette image indicating the region in the first input image where the object exists by inputting the first input image into a trained model without based on the difference between the first input image and a background image corresponding to the first input image; Image the region that does not include the aforementioned clay area. The second image processing apparatus includes: an acquisition means for acquiring a second input image obtained by imaging the object using a second imaging means different from the first imaging means; and a second generation means for generating a second silhouette image indicating the region in the second input image in which the object exists, based on the difference between the second input image and a background image in which the object does not exist, without using a trained model. [Effects of the Invention]

[0008] According to this disclosure, it is possible to generate silhouette images that can generate high-precision 3D shape data while suppressing the implementation cost of the image processing device. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an example of the functional configuration in an image processing system. [Figure 2] This block diagram shows an example of the hardware configuration in the first image processing unit, the second image processing unit, and the third image processing unit. [Figure 3] This figure shows an example of an image processing system application. [Figure 4] This flowchart shows an example of the flow of the determination process for the first or second imaging unit and the first or second image processing unit. [Figure 5] This figure shows an example of the field of view of an imaging device. [Figure 6] This flowchart shows an example of the processing flow for the first image processing unit, the second image processing unit, and the third image processing unit. [Figure 7] This flowchart shows an example of the flow of the determination process for the first or second imaging unit and the first or second image processing unit. [Figure 8] This is a diagram illustrating an example of an object's shadow and virtual image. [Figure 9] This figure shows examples of applying the image processing system to other imaging targets. [Modes for carrying out the invention]

[0010] The following descriptions of embodiments for implementing this disclosure will be made with reference to the drawings. The following embodiments are not limiting to this disclosure, and not all combinations of features described in these embodiments are essential to the solutions of this disclosure. The same components will be denoted by the same reference numerals. In addition, terms in which only the alphabet added after the reference numeral differ will indicate devices that have the same function but are different from each other. For example, the first imaging unit 110A and the first imaging unit 110B shown in Figure 1 are different devices that have the same function. Having the same function means that they each have at least a specific function, such as an imaging function, and for example, some of the functions and performance of the first imaging unit 110A and the first imaging unit 110B may differ from each other.

[0011] <Embodiment 1> In this embodiment, we will describe a case in which, in the process of separating the region corresponding to an object in an image (foreground region) from the background region using the background subtraction method, there is a region within the field of view of the imaging device in which the foreground region may not be detectable.

[0012] [System Configuration] Figure 1 is a block diagram showing an example of the functional configuration of the image processing system 100 according to Embodiment 1. The image processing system 100 generates three-dimensional shape data (hereinafter referred to as "object three-dimensional shape data") 160 that shows the three-dimensional shape of an object. The first imaging unit 110 images the object to be imaged, acquires the data of the image obtained by the imaging, and outputs the acquired image data to the first image processing unit 120. The first image processing unit 120 receives the image data output by the first imaging unit 110, separates the region corresponding to the object that is the target of the generation of three-dimensional shape data (object region) and the background region within the image, and generates a silhouette image of the object. The silhouette image data generated by the first image processing unit 120 is transmitted to the third image processing unit 150.

[0013] The first image processing unit 120 includes a first input unit 121, a first separation unit 122, and a transmission unit 124. The first input unit 121 receives the captured image data output by the first imaging unit 110 and stores the received captured image data in a storage device provided inside the first image processing unit 120. The first separation unit 122 processes the captured image to extract objects. to The foreground and background regions are separated to generate a silhouette image of the object. Specifically, the first separation unit 122 performs separation processing using a machine learning method to generate a silhouette image of the object. to The system separates regions to generate silhouette images. More specifically, the first separation unit 122 holds a trained model 123 internally. The trained model 123 is a model trained to take data from an captured image containing object regions as input and output data of an object's silhouette image. The trained model 123 generates silhouette images by separating the foreground and background regions in the captured image, for example, by performing a Semantic Segmentation task that performs pixel-by-pixel class identification of the input image.

[0014] As implementation methods of the learned model 123, there are various methods using CNN (Convolutional Neural Network), and there are a plurality of network structures using an Encoder layer, a Decorder layer, and a Skip structure. For example, the learned model 123 is realized by using a structure called SegNet or Unet. The data of the generated silhouette image and the data of the texture image that holds the color information of the object region in the silhouette image are output to the transmission unit 124. The transmission unit 124 receives the data of the silhouette image and the texture image output from the first separation unit 122 and transmits them to the third image processing unit 150. The transmission is performed via a communication I / F (interface) such as a LAN (Local Area Network) or a WAN (Wide Area Network).

[0015] The second imaging unit 130 images an imaging target, acquires data of the captured image obtained by the imaging, and outputs the acquired data of the captured image to the second image processing unit 140. The second image processing unit 140 receives the data of the captured image output from the second imaging unit 130, separates the region (object region) corresponding to the object that is the generation target of the three-dimensional shape data in the captured image and the background region, and generates a silhouette image of the object. The data of the silhouette image generated by the second imaging unit 130 is transmitted to the third image processing unit 150.

[0016] The second imaging unit 130 includes a second input unit 141, a second separation unit 142, and a transmission unit 144. The second input unit 141 stores the data of the received captured image in a storage device provided inside the second image processing unit 140. The second separation unit 142 separates the object region and the background region included in the captured image, and generates a silhouette image of the object. Specifically, the second separation unit 142 performs a separation process by background difference method to separate the object region and the background region included in the captured image, and generates a silhouette image of the object. More specifically, for example, the second separation unit 142 calculates the difference between a background image 143, which is a captured image obtained in a state where no object of the imaging target exists at the angle of view of the second imaging unit 130, and a captured image obtained in a state where an object exists. Further, the second separation unit 142 generates a silhouette image of the object by separating the object region and the background region in the captured image based on the calculated difference.

[0017] As a method for acquiring the background image 143 used in the separation process by background difference method, there is a method of using a captured image at the moment when no object exists as the background image. Also, a method of generating a background image using a plurality of captured images acquired over a certain period may be used. Specifically, for example, the change in pixel values in each captured image is observed in units of pixels or small regions, and when the change in pixel values is within a certain amount, the average value or the latest pixel value of the pixels within a certain period is used to generate the background image. The data of the silhouette image generated by the second separation unit 142 and the data of the texture image that holds the color information of the object region in the silhouette image are output to the transmission unit 144. The transmission unit 144 has the same function as the transmission unit 124, receives the data of the silhouette image and the texture image output from the second separation unit 142, and transmits these data to the third image processing unit 150.

[0018] The third image processing unit 150 generates three-dimensional shape data 160 of an object using silhouette image and texture image data received from the first image processing unit 120 and the second image processing unit 140. The third image processing unit 150 includes a receiving unit 151 and a shape generation unit 152. The receiving unit 151 receives silhouette image and texture image data transmitted from the first image processing unit 120 and the second image processing unit 140 and stores this data in a storage device provided inside the third image processing unit 150.

[0019] The shape generation unit 152 performs shape estimation and coloring processing to generate 3D shape data 160 using the silhouette image and texture image data received by the receiving unit 151. Specifically, the shape generation unit 152 first performs shape estimation processing, and then performs coloring processing to generate 3D shape data 160. For example, the view volume cross-section method can be used for shape estimation processing. For example, in the view volume cross-section method, the target space for generating 3D shape data is first tiled with unit volume rectangular parallelepipeds called voxels. Hereinafter, the set of voxels tiled in the target space for generating 3D shape data is called a voxel group. Next, each silhouette image generated by performing separation processing on the captured images of the first or second imaging units 110, 130 is projected onto the voxel group of the imaging range of the first or second imaging units 110, 130. Next, using all the silhouette images, uncolored 3D shape data is generated by removing voxels from this group that are not included in the area where the object region of each silhouette image is projected. Subsequently, as a coloring process, the texture image data corresponding to each silhouette image is used as color information, and the texture is applied to each voxel of the uncolored 3D shape data generated by the shape estimation process, thereby coloring the 3D shape data. By performing the above processes, the shape generation unit 152 generates 3D shape data 160.

[0020] Here, we will explain some cases in which the shape estimation process may fail. One case is when, for example, the silhouette image of an object does not include the region corresponding to the object (object region), meaning the object is not detected. Another case is when, for example, the silhouette image of an object includes a region other than the region corresponding to the object as the foreground region, meaning the object is falsely detected.

[0021] If an object is not detected, it means that the voxel corresponding to the object is mistakenly removed from the voxel group. Therefore, object non-detection must not occur in any of the images captured by the first and second imaging units 110 and 130. If an object is falsely detected, it will only result in the generation of false 3D shape data that differs from the shape of the object if the same image region corresponding to the same voxel is falsely detected in all of the images captured by the first and second imaging units 110 and 130. Therefore, in the case of false object detection, it is sufficient to suppress false detection in at least one of the images captured by the first or second imaging unit 110 or 130 that captures the imaging target space corresponding to the space in which the voxels constituting the false shape exist.

[0022] The 3D shape data 160 generated by the above process is a 3D point cloud, which is a collection of colored voxels, but the form of the 3D shape data is not limited to this. For example, the 3D shape data 160 may be 3D shape data consisting of a 3D polygon mesh generated from a colored 3D point cloud. In this case, for example, the shape generation unit 152 may include a process to generate 3D polygon mesh data from the 3D point cloud, which is the generated 3D shape data.

[0023] Figure 2 is a block diagram showing an example of the hardware configuration of the first image processing unit 120, the second image processing unit 140, and the third image processing unit 150. The hardware configuration of the first image processing unit 120, the second image processing unit 140, and the third image processing unit 150 (hereinafter collectively referred to as the "image processing device 200") is the same. The image processing device 200 includes a CPU 201, ROM 202, RAM 203, auxiliary storage device 204, display unit 205, operation unit 206, communication I / F 207, and bus 208.

[0024] The CPU 201 controls the entire image processing unit 200 using computer programs and data stored in ROM 202 or RAM 203, and implements each part of the image processing unit 200's functional configuration. The image processing unit 200 may have one or more dedicated hardware components separate from the CPU 201, and at least a portion of the processing performed by the CPU 201 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors). ROM 202 stores programs and other data that do not require modification. RAM 203 is used as a workspace for the CPU 201 and temporarily stores programs or data supplied from the auxiliary storage device 204, and data supplied from external sources via the communication interface 207. The auxiliary storage device 204 is composed of, for example, a large-capacity storage device such as a hard disk drive, and stores various types of data. RAM 203 and the auxiliary storage device 204 hold input data, intermediate data during processing, and output data for the image processing unit 200.

[0025] The RAM 203 or auxiliary storage device 204 of the first image processing unit 120 stores the data of the captured image received from the first imaging unit 110, the trained model 123 used by the first separation unit 122, and intermediate data from the separation process using the trained model 123. The RAM 203 or auxiliary storage device 204 of the first image processing unit 120 also stores the data of the silhouette image and texture image generated by the first separation unit 122. The RAM 203 or auxiliary storage device 204 of the second image processing unit 140 stores the data of the captured image received from the second imaging unit 130, the data of the background image 143 used by the second separation unit 142, and intermediate data for generating the background image 143. The RAM 203 or auxiliary storage device 204 of the second image processing unit 140 also stores the data of the silhouette image and texture image generated by the second separation unit 142. The RAM 203 or auxiliary storage device 204 of the third image processing unit 150 stores silhouette image and texture image data received from the first image processing unit 120 and the second image processing unit 140. In addition, the RAM 203 or auxiliary storage device 204 of the third image processing unit 150 stores intermediate data from the shape estimation process in the shape generation unit 152, as well as uncolored 3D shape data after the shape estimation process.

[0026] The display unit 205 is composed of a liquid crystal display or an LED (light-emitting diode), etc., and displays a GUI (Graphical User Interface) for the user to operate the image processing device 200. The operation unit 206 is composed of a keyboard, mouse, joystick, or touch panel, etc., and receives various instructions from the user and inputs them to the CPU 201. The CPU 201 operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206. The communication interface 207 is used for communication between the image processing device 200 and external devices.

[0027] The first image processing unit 120 has a communication interface 207, which includes an interface for receiving captured image data from the first imaging unit 110 and an interface for transmitting silhouette image and texture image data to the third image processing unit 150. The second image processing unit 140 has a communication interface 207, which includes an interface for receiving captured image data from the second imaging unit 130 and an interface for transmitting silhouette image and texture image data to the third image processing unit 150. The third image processing unit 150 has a communication interface 207, which includes an interface for receiving data transmitted from the first image processing unit 120 and the second image processing unit 140. The communication interface 207 is implemented by a wired network interface such as Ethernet, a wireless network interface such as Wireless LAN, or an SDI (Serial Digital Interface) interface for transmitting and receiving video signals. Bus 208 connects the various parts of the image processing device 200 as part of its hardware configuration and transmits information. In this embodiment, the display unit 205 and the operation unit 206 are located inside the image processing device 200, but at least one of the display unit 205 and the operation unit 206 may exist as a separate device outside the image processing device 200.

[0028] [Examples of system application] Figure 3 shows an example 300 of the application of the image processing system 100 according to Embodiment 1. The application example 300 shown in Figure 3 shows an example in which the image processing system 100 is applied to a soccer field as the imaging target. The imaging target space 301 contains a playing field 305 and objects 306 that are the targets for generating 3D shape data. In the case of soccer, objects 306 include objects 306A to 306C which are players, and object 306D which is a soccer ball. Of these, objects 306A and 306C are field players, and object 306B is the goalkeeper. Objects 306 may also include a referee, which is not shown in Figure 3. In addition, near the playing field 305, there are benches 307 where players wait, and signage 308 which is painted on the floor of the soccer field or laid out as a sign-like three-dimensional object. In addition, there are lighting devices 309 around the playing field 305 that are turned on when the illumination is insufficient, such as at night or in rainy weather.

[0029] The sign-like signage 308 also includes digital signage that displays content that changes over time. The lighting device 309 may be a light source that is locally positioned and emits strong light so that a dark shadow of object 306 is cast in a specific direction. Alternatively, the lighting device 309 may be a light source that is positioned at regular intervals around the competition field 305 so that a faint shadow of object 306 is cast evenly around it. Furthermore, when the floor surface of the competition field 305 is wet due to rain or the like, a virtual image of object 306 may be created on the floor surface due to specular reflection. The processing of the image processing system 100 when a shadow or virtual image of object 306 is created will be described in Embodiment 2.

[0030] Imaging devices 302A to 302P, which are either the first imaging unit 110 or the second imaging unit 130, are arranged around the imaging target space 301. Each imaging device 302 images at least a portion of the imaging target space 301. Figure 3 shows an arrangement of the imaging devices 302 surrounding the imaging target space 301 in a plan view from above, but the imaging devices 302 are also arranged with variations in their elevation angle relative to the imaging target space 301. This arrangement of imaging devices 302 makes it possible to image the object 306 from various angles, enabling the generation of high-quality 3D shape data.

[0031] Furthermore, in cases where a shadow or virtual image of object 306, as described in Embodiment 2, is generated, each imaging device 302 is determined to be either the first imaging unit 110 or the second imaging unit 130, depending on the elevation angle of the imaging device 302. The imaging device 302 outputs the captured image data to the imaging device control box 303, which is either the first image processing unit 120 or the second image processing unit 140. The imaging device control box 303 receives the captured image data and, if a region corresponding to object 306 (hereinafter referred to as the "region of object 306") exists in the captured image, generates silhouette image and texture image data of object 306. Furthermore, the imaging device control box 303 transmits the generated silhouette image and texture image data to the image processing server 304, which is the third image processing unit 150. The image processing server 304 generates three-dimensional shape data using the silhouette image and texture image data received from each imaging device control box 303.

[0032] Referring to Figures 4 and 5, the method for determining whether the imaging device 302 and the imaging device control box 303 are a first imaging unit 110 and a first image processing unit 120, or a second imaging unit 130 and a second image processing unit 140, will be explained. Figure 4 is a flowchart showing an example of the flow of the process for determining the first or second imaging unit 110, 130 and the first or second image processing unit 120, 140 according to Embodiment 1. The determination of the first or second imaging unit 110, 130 and the first or second image processing unit 120, 140 is made before imaging is performed using the image processing system 100, based on the field of view of each imaging device 302 determined when the imaging device 302 is installed. In the following explanation, the letter "S" at the beginning of the reference numeral means step (process). In S401, the user sets a specific area. In Embodiment 1, the specific region is a region in which, despite the presence of an object 306 region, object 306 may not be detected during the object 306 region separation process performed by the background subtraction method in the second image processing unit 140.

[0033] Referring to Figure 5, the specific region will be explained. Figure 5 is a diagram showing an example of the field of view of the imaging device 302 according to Embodiment 1. Figure 5(a) shows an example of the field of view 510 of the imaging device 302 that captures field player objects 306E and 306G, and ball object 306F, which are located on the playing field 305. In the field of view 510, all objects 306 move at a speed above a certain level, so the same object 306 does not remain in a pixel for a period of time or longer. Therefore, a background image without objects 306 can be generated. Here, it is assumed that in the field of view 510, there is a difference of a certain level or more between the color value of the object 306 and the pixel value of the background image. In this case, it is possible to create a background image, and the difference between the pixel value of the area of ​​object 306 in the image captured at the field of view 510 and the pixel value of the area corresponding to that area in the background image is also a certain level or more. Therefore, the field of view 510 of the imaging device 302 is considered not to include the specific region.

[0034] Figure 5(b) shows the field of view 520 of the imaging device 302 capturing field players (objects 306I and 306J), the ball (object 306H), and the goalkeeper (object 306K) located on the playing field 305. The field of view 520 of the imaging device 302 also includes the signage 308E painted on the floor of the soccer field. Figure 5(c) shows the same field of view 521 as the imaging device 302 shown in Figure 5(b), but after a certain period of time has elapsed from the state of field of view 520. The field of view 521 includes the objects 306 and signage 308 that were present in the state of field of view 520. Among these, objects 306 that have moved are indicated in Figure 5(c) by adding an apostrophe ('') to the end of their designation. For example, object 306I is the same field player as object 306I', indicating that the field player has moved.

[0035] Let's explain the specific region at a field of view of 520 (521). First, the goalkeeper object 306K hardly changes position even after a certain period of time has passed. In a background image generation method that determines pixels with no change in pixel value over a certain period of time as the background region, the region in the captured image corresponding to such a nearly stationary object 306K is determined to be a region that constitutes the background image. In other words, the region of object 306K is included in the background image. As a result, when object region separation processing is performed using background subtraction, no difference occurs between the captured image and the background image in the region of object 306K, resulting in object 306K not being detected. Therefore, a silhouette image of object 306K is not generated. In cases like this goalkeeper object 306K, the user can set regions where objects 306K with little movement, i.e., nearly stationary objects, may exist as specific regions.

[0036] On the other hand, the signage 308 painted on the floor of the soccer stadium is stationary and therefore included in the background image. Here, we consider the case where the difference between the pixel values ​​of the area corresponding to the signage 308 in the captured image and the pixel values ​​of the player's uniform, which is object 306, is smaller than a predetermined threshold. That is, we consider the case where the color of the signage 308 and the color of the uniform are similar. In this case, the difference between the pixel values ​​of the player's uniform and the signage 308 will not be greater than or equal to the predetermined threshold, resulting in the player's object 306 not being detected, and a silhouette image indicating the area corresponding to that player will not be generated. Thus, the user can set a background area in the background image that may contain a color similar to the color of object 306, which is the target of 3D shape data generation, as a specific area.

[0037] Figure 5(d) shows the field of view 530 of the imaging device 302 capturing field players (objects 306M and 306N), the ball (object 306L), and the bench (307C) where the reserve players are waiting, all located on the playing field 305. Here, neither the bench (307C) nor the reserve players within it are to be considered targets for generating 3D shape data. Since the reserve players on the bench (307C) are natural people, they move over time. When generating a background image at such a field of view 530 of the imaging device 302, some of the regions in the captured image corresponding to the reserve players will not be pixels that remain unchanged for a certain period of time or longer, and therefore will not be included in the background image. If object region separation processing is performed on the captured image using background subtraction using such a background image, some of the regions corresponding to the reserve players will be separated as foreground regions, and a silhouette image showing these foreground regions will be generated.

[0038] On the other hand, some areas in the captured image corresponding to stationary substitute players will have pixels that remain unchanged for a certain period of time or longer, and will therefore be incorporated into the background image. In this case, there will be no significant difference between the pixel values ​​of the areas corresponding to the substitute players' bodies or uniforms incorporated into the background image and the pixel values ​​of the areas corresponding to the field player's body or uniform (object 306) in the captured image. As a result, the object areas cannot be properly separated, some parts of object 306 will not be detected, and a silhouette image will be generated in which part of the area corresponding to object 306 is missing. In this way, a stable background image cannot be generated, and background areas in the background image that may contain colors similar to the color of object 306 can be designated as specific areas by the user.

[0039] Figure 5(e) shows the field of view 540 of the imaging device 302 capturing field players (objects 306P and 306Q) and the ball (object 306O) located on the playing field 305. The field of view 540 of the imaging device 302 also includes a signboard-like digital signage display (signage 308F) installed outside the playing field 305. Figure 5(f) shows the same field of view 541 as in Figure 5(e), but after a certain period of time has elapsed. The field of view 541 includes both object 306 and signage 308, which were present in the state of the imaging device 302's field of view 540. Objects 306 and signage 308 that have moved are indicated in Figure 5(f) by adding an apostrophe ('') to the end of their designations. For example, the digital signage display 308F is designated as 'Signage 308F' because its displayed content has changed.

[0040] The following describes a specific area at a field of view of 540 (541). Because the display content of the digital signage 308 changes, the pixels in the area corresponding to the signage 308 in the captured image will not be pixels whose pixel value remains unchanged for a certain period of time or longer, and this area will not be included in the background image. Therefore, when object area separation processing is performed on the captured image at a field of view of 540 and the captured image at a field of view of 541 using background subtraction, the area of ​​the signage 308F will also be mistakenly separated as a foreground area. As a result, a silhouette image of the signage 308F, which is not the target of 3D shape data generation, is generated.

[0041] On the other hand, if the pixel values ​​of the area corresponding to the signage 308 in the captured image do not change for a certain period of time or longer, that area will be incorporated into the background image. In this case, if the colors used in the display content of the digital signage when incorporated into the background image are similar to the colors of object 306, object 306 will not be detected. As a result, a silhouette image of object 306 will not be generated. Users can set as specific areas areas in which a silhouette image of an object that is not the target of 3D shape data generation may be generated, and areas in which a stable background image cannot be generated and object 306 may not be detected.

[0042] In summary, the specific region in Embodiment 1 includes a region where objects with little movement may exist, and a region where a fixed background with a color similar to the object's color may exist. Furthermore, the specific region in Embodiment 1 also includes a region where there is movement, and a part of the background image may contain a background area with a color similar to the object 306's color. Note that the specific region is not limited to those described above; a region containing factors that cause objects to be undetected during the object region separation process using background subtraction may also be set as the specific region. By setting such a specific region, the user can apply this disclosure to cases other than those described above with reference to Figure 5.

[0043] Let's return to the explanation of the flowchart shown in Figure 4. After S401, the user decides whether to process all imaging devices 302 using the first imaging unit 110 or the second imaging unit 130, and whether to process the captured image data output by the imaging devices 302 using the first image processing unit 120 or the second image processing unit 140. Specifically, the user performs steps S403 to S405 for each imaging device 302. Specifically, for example, in the application example 300 shown in Figure 3, the user performs steps S403 to S405 for all imaging devices 302 from imaging device 302A to imaging device 302P. Note that S402 indicates the start of the loop of steps S403 to S405.

[0044] In S402, the user selects any imaging device 302 from among one or more imaging devices 302 that have not been selected so far. Next, in S403, the user determines whether the field of view captured by the imaging device 302 selected in S402 includes a specific region. If it is determined in S403 that it includes a specific region, in S404 the user determines that the imaging device 302 selected in S402 is the first imaging unit 110 and the imaging device control box 303 corresponding to that imaging device 302 is the first image processing unit 120. If it is determined in S403 that it does not include a specific region, in S405 the user determines that the imaging device 302 selected in S402 is the second imaging unit 130 and the imaging device control box 303 corresponding to that imaging device 302 is the second image processing unit 140.

[0045] S406 indicates the end of the loop of steps S403 to S405. In S406, the user determines whether or not all imaging devices 302 were selected in S402. If in S406 the user determines that not all imaging devices 302 have been selected, i.e., that there are imaging devices 302 that have not been selected, the user returns to S402 and selects any imaging device 302 that has not been selected so far. Subsequently, the steps from S402 to S406 are repeated until the user determines in S406 that all imaging devices 302 have been selected. If the user determines in S406 that all imaging devices 302 have been selected, the user terminates the decision process shown in the flowchart of Figure 4.

[0046] Referring to the field of view angles 510, 520, 530, and 540 shown in Figure 5, a specific example of how to determine the imaging device 302 and the imaging device control box 303 will be described. Since the field of view angle 510 of the imaging device 302 does not include the specific region, it is determined that the imaging device 302 is the second imaging unit 130, and the imaging device control box 303 connected to the imaging device 302 is the second image processing unit 140. On the other hand, since the field of view angles 520, 530, and 540 of the imaging device 302 all include the specific region, it is determined that the imaging device 302 is the first imaging unit 110, and the imaging device control box 303 connected to the imaging device 302 is the first image processing unit 120.

[0047] For example, the user first places all the imaging devices 302 in the desired positions and adjusts the field of view of each imaging device 302 to the desired field of view. Subsequently, based on the determination in the decision process shown in the flowchart of Figure 4, the user places the first image processing unit 120 near, for example, the imaging device 302 that has been determined to be the first imaging unit 110, and connects the first image processing unit 120 to the first imaging unit 110 and the third image processing unit 150. Similarly, the user places the second image processing unit 140 near, for example, the imaging device 302 that has been determined to be the second imaging unit 130, and connects the second image processing unit 140 to the second imaging unit 130 and the third image processing unit 150.

[0048] Referring to Figure 6, the operation of the first image processing unit 120, the second image processing unit 140, and the third image processing unit 150 will be described. Figure 6(a) is a flowchart showing an example of the processing flow of the first image processing unit 120 according to Embodiment 1. First, at S601, the first input unit 121 receives the captured image data output by the first imaging unit 110. Next, at S602, the first separation unit 122 inputs the captured image data received at S601 to the trained model 123. Next, at S603, the first separation unit 122 acquires the silhouette image data generated by the trained model 123. Next, at S604, the transmission unit 124 outputs the silhouette image data acquired at S603 to the third image processing unit 150. After S604, the first image processing unit 120 completes the processing shown in the flowchart in Figure 6(a), and repeatedly executes the processing shown in the flowchart in Figure 6(a) each time the first imaging unit 110 outputs new captured image data.

[0049] Figure 6(b) is a flowchart showing an example of the processing flow of the second image processing unit 140 according to Embodiment 1. First, at S611, the second input unit 141 receives the captured image data output by the second imaging unit 130. Next, at S612, the second separation unit 142 generates a silhouette image by performing a background subtraction method separation process on the captured image data received at S601 using the background image data. Next, at S613, the transmission unit 144 outputs the silhouette image data generated at S612 to the third image processing unit 150. After S613, the second image processing unit 140 completes the processing shown in the flowchart in Figure 6(b), and repeatedly executes the processing shown in the flowchart in Figure 6(b) each time the second imaging unit 130 outputs new captured image data.

[0050] Figure 6(c) is a flowchart showing an example of the processing flow of the third image processing unit 150 according to Embodiment 1. First, at S621, the receiving unit 151 receives silhouette image and texture image data transmitted from the first image processing unit 120 and the second image processing unit 140. Here, the first imaging unit 110 and the second imaging unit 130 perform imaging in sync with each other, for example, and the receiving unit 151 receives silhouette image and texture image data based on the data of the images obtained by imaging at the same time. Note that the same time here is not strictly limited to the same time, but includes approximately the same time.

[0051] After S621, in S622, the shape generation unit 152 performs shape estimation processing of 3D shape data using the silhouette image data received in S621. Next, in S623, the shape generation unit 152 performs coloring processing on the 3D shape data generated in S622 using the texture image data to generate 3D shape data 160. After S623, the third image processing unit 150 completes the processing shown in the flowchart in Figure 6(c). Subsequently, the third image processing unit 150 repeatedly executes the processing shown in the flowchart in Figure 6(c) each time the first image processing unit 120 and the second image processing unit 140 output new silhouette image and texture image data.

[0052] As described above, the image processing system 100 is configured to set a specific region as a region where object undetection may occur using the background subtraction method. Furthermore, the image processing system 100 is configured so that the data of the captured image output from the imaging device 302 whose field of view includes the specific region is input to an image processing device that performs object region separation using the trained model 123. Moreover, the image processing system 100 is configured so that the data of the captured image output from the imaging device 302 whose field of view does not include the specific region is input to an image processing device that performs object region separation using the background subtraction method. Furthermore, the image processing system 100 is configured to determine which image processing device will appropriately perform object region separation according to the field of view of the imaging device 302. With the image processing system 100 configured as described above, it is not necessary to have a single image processing device that can perform both background subtraction method separation and machine learning method separation. Therefore, it is possible to generate silhouette images that can generate high-precision 3D shape data while suppressing the implementation cost of the image processing device.

[0053] <Embodiment 2> In this embodiment, we will describe a case in which, during object region separation processing using background subtraction, there is a possibility that an area other than the area of ​​object 306 may be mistakenly detected as the foreground area within the field of view of the imaging device 302. Areas that may be mistakenly detected include areas where the shadow of object 306 may be generated, and areas where the image of object 306 may be reflected by the surface of the competition field 305, resulting in a virtual image. The functional configuration of the image processing system according to Embodiment 2 is the same as the functional configuration of the image processing system 100 shown in Figure 1, so the differences from Embodiment 1 will be described below.

[0054] [System Application Examples] As an example of application, it will be described as being the same as the example of application of the image processing system shown in Figure 3, similar to Embodiment 1. Referring to Figures 7 and 8, the method for determining whether the imaging device 302 and the imaging device control box 303 are a first imaging unit 110 and a first image processing unit 120, or a second imaging unit 130 and a second image processing unit 140 will be explained. Figure 7 is a flowchart showing an example of the flow of the process for determining the first or second imaging unit 110, 130 and the first or second image processing unit 120, 140 according to Embodiment 2.

[0055] The decision process shown in the flowchart of Figure 7 is performed before the image processing system starts, based on the field of view determined when the imaging device 302 is installed and the weather or lighting conditions when the image processing system is used. Changes in weather or lighting conditions and the resulting changes in shadows or virtual images of objects 306 that may occur in the field of view of each imaging device 302 are assumed to have been investigated in advance. Note that the flowchart of Figure 7 only shows whether a certain imaging device 302 is the first imaging unit 110 or the second imaging unit 130. However, the imaging device control box 303 is also determined to be either the first image processing unit 120 or the second image processing unit 140 in accordance with the determination of whether it is the first imaging unit 110 or the second imaging unit 130. Specifically, if a certain imaging device 302 is determined to be the first imaging unit 110, the imaging device control box 303 corresponding to that imaging device 302 is determined to be the first image processing unit 120. On the other hand, if it is decided that a certain imaging device 302 is the second imaging unit 130, then the imaging device control box 303 corresponding to that imaging device 302 is determined to be the second image processing unit 140.

[0056] First, in S701, the user sets a specific area. In Embodiment 2, the specific area is the area where a shadow or virtual image of object 306 may occur. Next, in S702, the user determines whether or not a dark shadow will occur in a specific direction. If in S702 the user determines that a dark shadow will occur in a specific direction, in S703 the user selects one or more imaging devices 302 capable of imaging the area where a dark shadow may occur as the first imaging unit 110. Since each imaging device 302 captures the entire field while having overlapping areas in their fields of view, there are multiple imaging devices 302 whose fields of view include the area where a dark shadow may occur. The user selects one or more of these as the first imaging unit 110.

[0057] Figure 8 is a diagram illustrating an example of the shadow and virtual image of object 306 according to Embodiment 2. Figure 8(a) shows an example of a shadow 811 generated in the field of view 810 of the imaging device 302. In the example shown in Figure 8(a), the shadow 811 of object 306R is generated as a dark shadow on the playing field 305 surface due to strong light from a specific direction by the lighting device 309C. Figure 8(b) shows an example of an overhead view image 812 obtained by virtually viewing Figure 8(a) from directly above. When a dark shadow 811 is generated as shown in Figure 8(b), the shadow 811 is generated in a specific direction relative to object 306. Figures 8(a) and (b) show an example where the shadow 811 is generated in one direction, but depending on the number and arrangement of lighting devices, the shadow of object 306 may be generated in two or more directions. However, when the shadow of object 306 is generated in two or more directions, the subsequent shadows will not be dark shadows. This is because, from the perspective of object 306, the area of ​​the competition field 305 in the direction of the lighting device that emits strong light is illuminated by the light from that device, and therefore the shadows cast in that area are not dark shadows.

[0058] If it is anticipated that a dark shadow 811 will occur, as shown in the overhead image 812 of Figure 8(b), the user designates a portion of one or more imaging devices 302 located in the direction from which the dark shadow 811 is expected to occur, as the first imaging unit 110. Since it is sufficient to delete the voxels corresponding to the area where the dark shadow 811 occurs, if there is an imaging device 302 that captures the entire dark shadow 811 within its field of view, only that imaging device 302 can be designated as the first imaging unit 110 as a portion of the above. In actual imaging, it is rare for conditions to be met such that a single imaging device 302 can completely delete the voxels corresponding to the area where a dark shadow may occur. In such cases, a sufficient number of imaging devices 302 are designated as the first imaging unit 110 to delete the voxels corresponding to the area where a dark shadow may occur across the entire competition field 305.

[0059] After S703, or if it is determined in S702 that no dense shadow is generated in a particular direction, in S704 the user determines whether the shadow of object 306 is evenly distributed around object 306. If it is determined in S704 that the shadow is evenly distributed around object 306, the user performs the process in S705. Specifically, in S705 the user determines a predetermined number of imaging devices 302 from all imaging devices 302 capable of imaging at least a portion of the area where the shadow can be evenly distributed around object 306 as the first imaging unit 110. This is to suppress any remaining voxels due to the shadow that is evenly distributed around object 306.

[0060] Figure 8(c) shows an example of a shadow 821 evenly distributed around an object 306 within the field of view 820 of the imaging device 302. In the example shown in Figure 8(c), the shadow 821 is generated around the object 306S by the light emitted by numerous lighting devices 309 surrounding the competition field 305. Figure 8(d) shows an example of an overhead view image 822, which is a virtual view of Figure 8(c) from directly above. As shown in Figure 8(d), if a shadow 821 can be evenly distributed around the object 306, the user determines a predetermined number of imaging devices 302 from among all imaging devices 302 whose field of view includes at least a portion of the area where the shadow 821 can be generated as the first imaging unit 110.

[0061] The predetermined number of imaging devices is determined, for example, by the angle difference between the optical axis of each imaging device 302 and the object 306 or the space to be imaged. The user determines a predetermined combination of imaging devices 302 that can remove voxels corresponding to the area of ​​shadow 821 generated around the object 306S, and determines the imaging devices 302 included in the determined combination as the first imaging unit 110. Specifically, for example, the user determines two or three imaging devices 302 selected from a plurality of imaging devices 302 that image a part of the space to be imaged, such that the angle difference between the optical axes of the imaging devices 302 is 120 to 180 degrees, as the first imaging unit 110. With the two or three imaging devices 302 selected in this way, the shadow 821 can be imaged without being obstructed by the object 306S.

[0062] If, after S705, or if in S704, it is determined that shadows are not evenly distributed around the object 306, then in S706, the user determines whether or not a virtual image is generated around the object 306. If, in S706, it is determined that a virtual image is generated around the object 306, then in S707, the user designates a portion of one or more imaging devices 302 capable of imaging the region where the virtual image is generated as the first imaging unit 110.

[0063] Figure 8(e) shows an example of a virtual image 831 of object 306 generated at the field of view 830 of the imaging device 302. In the example shown in Figure 8(e), the virtual image 831 is generated by the reflection of the image of object 306T from the surface of the competition field 305. Figure 8(f) is a cross-sectional image 832 of Figure 8(e) viewed virtually from the side. The virtual image of object 306 is generated on the line where the plane determined by the line connecting the imaging device 302 and the object 306 and a vector perpendicular to the surface of the competition field 305 intersects with the surface of the competition field 305. Therefore, the location where the virtual image is generated differs for each imaging device 302. For example, in the cross-sectional image 832, the virtual image generated by the reflection of the image of object 306T from the field surface 833 appears as virtual image 834 in imaging device 302Q and as virtual image 835 in imaging device 302R. Therefore, the false 3D shape data due to the virtual image is generated in the space where the 3D shape data is generated, and the field surface 8 It does not occur in the area corresponding to the upper part of 33.

[0064] However, the target space for generating 3D shape data is the field surface 8 If the region corresponding to the area below 33 is set, false 3D shape data due to virtual images will be generated in the following space within the 3D shape data generation target space. Specifically, in this case, the field plane of the region where object 306 exists within the 3D shape data generation target space. 8 In the space corresponding to the region that is symmetrical with respect to 33, false 3D shape data is generated by virtual images. For example, in the cross-sectional image 832, 3D shape data 836 is generated as false 3D shape data due to virtual images. Therefore, the user is unable to see the field surface. 8 A sufficient number of imaging devices 302 are selected as the first imaging unit 110 to completely remove the voxels of false 3D shape data that occur in the space corresponding to the lower region of 33.

[0065] Unlike the shadow of object 306, the virtual image of object 306 is not obscured by object 306. Therefore, basically, the user only needs to select any one of the multiple imaging devices 302 that image the area where virtual images may occur as the first imaging unit 110 of the imaging device 302. However, if multiple objects 306 are densely packed together, the virtual image of one object 306 may be hidden by another object 306. Taking such cases into consideration, the user may select two or more imaging devices 302 from the multiple imaging devices 302 that image the area where virtual images may occur as the first imaging unit 110.

[0066] If, after S707, or if it is determined in S706 that no virtual image is generated around the object 306, then in S708, the user determines whether or not a shadow is generated in the area surrounded by multiple objects 306. If it is determined in S708 that a shadow is generated, then in S709, the user selects a portion of one or more imaging devices 302 whose optical axis elevation angle is greater than or equal to a predetermined angle as the first imaging unit 110. By selecting an imaging device 302 whose optical axis elevation angle is greater than or equal to a predetermined angle as the first imaging unit 110, it is possible to suppress the incomplete removal of 3D shape data due to shadows generated in the area surrounded by multiple objects 306.

[0067] Figure 8(g) shows an example of a shadow 841 generated in the area surrounded by objects 306 at a field of view 840 of the imaging device 302. In the example shown in Figure 8(g), the shadow 841 is generated in the area surrounded by objects 306U, 306V, 306W, and 306X due to light irradiated by numerous lighting devices 309 surrounding the competition field 305. Figure 8(h) shows an example of a cross-sectional image 842 virtually viewed from the side of Figure 8(g). Voxels corresponding to the areas where shadows 841 exist, as shown in Figures 8(g) and 8(h), can be removed using, for example, a silhouette image corresponding to the image captured by the imaging device 302, as shown below. For example, as shown in the cross-sectional image 842, the imaging device 302 has an optical axis elevation angle greater than the elevation angle 843 of the tangent line drawn from the field plane at the center of the densely packed multiple objects 306 to the top of object 306U.

[0068] For example, the user sets such an elevation angle as an angle threshold based on the characteristics of the object 306 to be imaged or the sport, and determines a portion of one or more imaging devices 302 whose optical axis elevation angle is greater than or equal to this angle threshold as the first imaging unit 110. Specifically, the user selects two or three imaging devices 302 that satisfy the above-mentioned conditions for imaging a portion of the image target space, similar to the case of a shadow 821 or a virtual image 831, where the angle difference of the optical axes between the imaging devices 302 is between 120 and 180 degrees. Furthermore, the user determines the selected two or three imaging devices 302 as the first imaging unit 110.

[0069] If, after S709, it is determined in S708 that no shadow will be generated, then in S710, the user determines whether or not a shadow will be generated on the lower part of the object 306. If, in S710, it is determined that a shadow will be generated on the lower part of the object 306, then in S711, the user designates a portion of one or more imaging devices 302 whose elevation angle is less than or equal to a predetermined angle as the first imaging unit 110. By designating a portion of one or more imaging devices 302 whose elevation angle is less than or equal to a predetermined angle as the first imaging unit 110, it is possible to suppress the incomplete removal of 3D shape data due to the shadow generated on the lower part of the object 306.

[0070] Figure 8(i) shows an example of a shadow 847 that occurs at the bottom of object 846. Figure 8(i) shows a cross-sectional image 844 of object 846 viewed from the horizontal. Here, we will use the shadow 847 generated by a table-like object 846, rather than the object 306 used in the previous explanation. The voxel corresponding to the area where the shadow 847 that forms at the bottom of the table-like object 846 occurs can be removed using a silhouette image corresponding to the image captured by the imaging device 302 as follows. For example, as shown in the cross-sectional image 844, the imaging device 302 has an elevation angle of the optical axis that is smaller than the elevation angle 848 of the tangent line drawn from the field plane at the center of object 846 to the edge of the top surface of object 846.

[0071] For example, the user sets such an elevation angle as an angle threshold based on the shape of the object 846 to be imaged, and determines a portion of one or more imaging devices 302 whose optical axis elevation angle is less than or equal to this angle threshold as the first imaging unit 110. The user selects two or three imaging devices 302 from among a plurality of imaging devices 302 that satisfy the above conditions for imaging a portion of the image target space, where the angle difference of the optical axes between the imaging devices 302 is between 120 and 180 degrees, similar to the case of shadows 821, virtual images 831, and shadows 841. Furthermore, the user determines the two or three selected imaging devices 302 as the first imaging unit 110.

[0072] If, after S711, or if in S710, it is determined that no shadow will be cast on the lower part of object 306, then in S712, the user determines that the remaining imaging devices 302 will be the second imaging unit 130. Specifically, the user determines that all imaging devices 302 that were not determined to be the first imaging unit 110 in the previous steps will be the second imaging unit 130.

[0073] As described above, the image processing system 100 is configured to set a specific region as a region where false detection of object regions may occur in the background subtraction method. Here, the specific region in Embodiment 2 is a region where the shadow or virtual image of object 306 may occur. Furthermore, the image processing system 100 is configured so that the data of captured images output from at least a portion of one or more imaging devices 302 whose field of view includes the specific region is input to an image processing device that performs object region separation using the trained model 123. Moreover, the image processing system 100 is configured so that the data of captured images output from imaging devices 302 whose field of view does not include the specific region is input to an image processing device that performs object region separation using the background subtraction method. With the image processing system 100 configured as described above, it is not necessary to enable both background subtraction and machine learning separation processing in a single image processing device, and it is possible to generate silhouette images that can generate highly accurate 3D shape data.

[0074] <Other Embodiments> In Embodiments 1 and 2, the application of the image processing system 100 to a soccer game was described as an example, but the application of the image processing system 100 is not limited to this. Figure 9 shows examples of the application of the image processing system 100 to other imaging targets. Figure 9 shows an example of factors that may cause objects to be undetected or falsely detected in the object region separation process using the background subtraction method for each imaging target. The region containing these factors is set as a specific region using the same setting method as in Embodiments 1 and 2. Furthermore, based on the specific region and the field of view of the imaging device 302, the imaging device 302 is determined to be either the first imaging unit 110 or the second imaging unit 130. This makes it possible to generate a silhouette image that can produce highly accurate 3D shape data without any missing or un-etched parts of the shape of the object 306.

[0075] Furthermore, Embodiment 1 described, as an example, a case in which factors cause object failure to occur during object region separation processing using background subtraction. Furthermore, Embodiment 2 described, as an example, a case in which factors cause false detection of objects during object region separation processing using background subtraction. In each embodiment, examples of cases where factors cause object failure and false detection are shown separately, but the image processing system 100 can also be applied when these factors coexist. In this case, first, the imaging device 302 in which factors causing failure to occur are determined as the first imaging unit 110 using the determination method shown in Embodiment 1. Subsequently, using the determination method shown in Embodiment 2, a portion of the remaining imaging devices 302 in which factors causing false detection occur are determined as the first imaging unit 110, and further, the remaining imaging devices 302 are determined as the second imaging unit 130. By determining the first imaging unit 110 and the second imaging unit 130 in this way, it is possible to generate a silhouette image that can generate highly accurate 3D shape data without chipping or uncut areas for the shape of the object 306.

[0076] Furthermore, this disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to each device of the image processing system 100 via a network or storage medium, and by having one or more processors in those devices read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0077] Furthermore, within the scope of this disclosure, it is possible to freely combine each embodiment, modify any component of each embodiment, or omit any component in each embodiment.

[0078] <Structure of this disclosure> This disclosure includes the following configuration and method:

[0079] [Configuration 1] A first image processing device generates data for a first silhouette image indicating the region in which the object exists in the first input image by inputting the image data obtained by imaging an object using a first imaging means that images a region including at least a part of a specific region as data for a first input image into a trained model. A second image processing device generates data for a second silhouette image indicating the region in the second input image where the object exists, by calculating the difference between the second input image and a background image which is an image obtained by imaging by the second imaging device when the object does not exist in the region imaged by the second imaging device, using the image data obtained by imaging the object by a second imaging device different from the first imaging device as the data for the second input image. An image processing system characterized by having the following features.

[0080] [Configuration 2] The second imaging means images an area that does not include the specific area. The image processing system described in Configuration 1, characterized by the above.

[0081] [Configuration 3] The aforementioned specific region is a region in which the object can exist in a substantially stationary state. An image processing system according to configuration 1 or 2, characterized by the above.

[0082] [Structure 4] The specified region is a region in which the difference between the color of the object and the background color of the region captured by the imaging device, which is the first imaging means or the second imaging means, is smaller than a predetermined standard. An image processing system according to any one of configurations 1 to 3 characterized by the above.

[0083] [Composition 5] The aforementioned specific region is a region where a shadow of the object or a virtual image may be generated due to the reflection of the object's image in a part of the area captured by the imaging device, which is the first imaging means or the second imaging means. An image processing system according to any one of configurations 1 to 4 characterized by the above.

[0084] [Composition 6] The first imaging means is a part of one or more imaging devices whose field of view includes at least a part of the specified area. An image processing system according to any one of configurations 1 to 5, characterized by the above.

[0085] [Composition 7] The first imaging means is set to be one of two or more imaging devices whose field of view includes at least a portion of the specified region, such that the angle between the optical axis vectors of the imaging devices is greater than or equal to a predetermined angle. An image processing system according to any one of configurations 1 to 6 characterized by the above.

[0086] [Structure 8] The first imaging means is one or more imaging devices whose field of view includes at least a portion of the specified region, wherein the angle between the optical axis vector of the imaging device and the field surface of the object to be imaged by the imaging device is greater than or equal to a predetermined angle or less than or equal to a predetermined angle. An image processing system according to any one of configurations 1 to 7, characterized by the above.

[0087] [Composition 9] The first image processing apparatus is A first acquisition means that acquires the image data obtained by imaging the object with the first imaging means as the data of the first input image, A first generation means generates data of the first silhouette image by inputting the acquired data of the first input image into the trained model, A first output means for outputting the generated first silhouette image, It has, The second image processing device is A second acquisition means acquires the image data obtained by imaging the object using the second imaging means as the data of the second input image, A second generation means generates data for the second silhouette image by calculating the difference between the acquired second input image and the background image, A second output means for outputting the generated second silhouette image, Having An image processing system according to any one of configurations 1 to 8, characterized by the above.

[0088] [Configuration 10] A third image processing device generates three-dimensional shape data indicating the shape of the object using the data of the first silhouette image generated by the first image processing device and the data of the second silhouette image generated by the second image processing device. Having further, An image processing system according to any one of configurations 1 to 9, characterized by the above.

[0089] [Composition 11] The third image processing apparatus is Silhouette acquisition means for acquiring the data of the first silhouette image output by the first image processing device and the data of the second silhouette image output by the second image processing device, A shape generation means that generates the three-dimensional shape data using the acquired data of the first silhouette image and the data of the second silhouette image, Having An image processing system as described in configuration 10, characterized by the above.

[0090] [method] A first image processing step in a first image processing apparatus, comprising: a first acquisition step of acquiring image data obtained by imaging an object using a first imaging means that images a region including at least a part of a specific region, as data of a first input image; a first step of generating data of a first silhouette image indicating the region in the first input image where the object exists by inputting the data of the first input image into a trained model; and a first output step of outputting the data of the first silhouette image; A second image processing step in a second image processing apparatus, comprising: a second acquisition step of acquiring image data obtained by imaging the object with a second imaging means different from the first imaging means as data for a second input image; a second generation step of generating data for a second silhouette image indicating the region in the second input image where the object exists by calculating the difference between the second input image and a background image which is an image obtained by imaging by the second imaging means when the object is not present in the region imaged by the second imaging means; and a second output step of outputting the data for the second silhouette image, An image processing method characterized by including [a certain element]. [Explanation of symbols]

[0091] 100 Image Processing Systems 110 First Imaging Unit 120 First Image Processing Unit 121 First Input Section 122 1st separation section 123 Pre-trained models 124 Transmission section 130 Second Imaging Unit 140 Second Image Processing Unit 141 Second Input Section 142 Second separation section 143 Background Images 144 Transmission section

Claims

1. A first image processing apparatus comprising: an acquisition means for acquiring a first input image obtained by imaging an object using a first imaging means for imaging an area including the clay portion of a baseball field; and a first generation means for generating a first silhouette image indicating the area in the first input image in which the object exists by inputting the first input image into a trained model without relying on the difference between the first input image and a background image corresponding to the first input image; A second image processing apparatus comprising: an acquisition means for acquiring a second input image obtained by imaging the object using a second imaging means different from the first imaging means, which images a region that does not include the clay area; and a second generation means for generating a second silhouette image showing the region in the second input image where the object exists, based on the difference between a background image in which the object does not exist and the second input image, without using a trained model; An image processing system characterized by having the following features.

2. The system further comprises a third image processing device that generates three-dimensional shape data indicating the shape of the object using the first silhouette image and the second silhouette image, The first image processing device has output means for outputting the first silhouette image to the third image processing device, The second image processing device has output means for outputting the second silhouette image to the third image processing device. The image processing system according to claim 1, characterized by the following:

3. The third image processing device is Silhouette acquisition means for acquiring the first silhouette image output by the first image processing device and the second silhouette image output by the second image processing device, A shape generation means that generates the three-dimensional shape data using the acquired first silhouette image and the second silhouette image, Having further The image processing system according to claim 2, characterized by the following:

4. A first image processing step in a first image processing apparatus, comprising: an acquisition step of acquiring a first input image obtained by imaging an object using a first imaging means that images an area including the clay portion of a baseball field; and a first generation step of generating a first silhouette image indicating the area in the first input image where the object exists by inputting the first input image into a trained model without relying on the difference between the first input image and a background image corresponding to the first input image; A second image processing step in a second image processing apparatus, comprising: an acquisition step of acquiring a second input image obtained by imaging the object using a second imaging means different from the first imaging means, which images a region that does not include the clay area; and a second generation step of generating a second silhouette image showing the region in the second input image where the object exists, based on the difference between a background image in which the object does not exist and the second input image, without using a trained model; An image processing method characterized by including

Images