Image processing device, image processing metho and program

Abstract

To determine consistency between information inputted during image generation and the generated image.SOLUTION: An image processing device includes input means for accepting input of joint information of a generation object, generation means for generating an image of the generation object on the basis of the joint information, detection means for detecting joints from a generation image generated by the generation means, shield determination means for determining shielded states of the joints in the generation image, and consistency determination means for determining consistency between the joint information and the generation image on the basis of detection results of the joints and the shielded states of the joints.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for determining consistency between information input when generating an image and the generated image. [Background technology]

[0002] Generative AI (artificial intelligence) is known that automatically generates images based on input information such as descriptive text and joint information of people, animals, etc. (Non-Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-9693 [Non-patent literature]

[0004] [Non-Patent Document 1] Rombach, Robin, et al. ``High-resolution image synthesis with latent diffusion models.'' Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022. Summary of the Invention [Problem to be solved by the invention]

[0005] When generating images of people, animals, etc. using image generation AI, the generated images may have specific body parts (arms, legs, fingers, etc.) added or missing. Patent Document 1 describes a method of storing pattern information related to the posture and facial expression of people, animals, etc. in advance and determining whether the subject included in the image matches the pattern information.

[0006] However, in the privileged document 1, the consistency between the information input at the time of image generation and the generated image is not determined.

[0007] The present invention has been made in view of the above-mentioned problems, and its object is to realize a technique for determining consistency between information input at the time of image generation and the generated image. [Means for solving the problem]

[0008] In order to solve the above problems and achieve the object, the image processing device of the present invention has an input means for inputting joint information of an object to be generated, a generation means for generating an image of the object to be generated based on the joint information, a detection means for detecting joints from the generated image generated by the generation means, an occlusion determination means for determining an occlusion state of the joint in the generated image, and a consistency determination means for determining consistency between the joint information and the generated image based on the detection result of the joint and the occlusion state of the joint. [Effects of the Invention]

[0009] According to the present invention, it is possible to determine the consistency between information input at the time of image generation and the generated image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating an example of an apparatus configuration according to an embodiment of the present invention. [Figure 2] 3A to 3C are diagrams illustrating input information and generated images when generating an image according to the present embodiment. [Figure 3] 10 is a flowchart illustrating a consistency determination process according to the present embodiment. [Figure 4] FIG. 2 is a block diagram illustrating an example of a device configuration including a joint information generating unit according to the present embodiment. [Figure 5] FIG. 4 is an explanatory diagram of joint information according to the present embodiment. [Figure 6] FIG. 1 is a block diagram illustrating an example of a device configuration including an image selection unit according to the present embodiment. [Figure 7]FIG. 10 is a block diagram illustrating an example of the configuration of an apparatus for performing occlusion determination based on joint information and generated images according to the present embodiment. [Figure 8] 10 is a flowchart illustrating an example of an occlusion determination process according to the present embodiment. [Figure 9] 10A to 10C are diagrams illustrating a method for matching the posture of a generated image with the posture of a three-dimensional planar model in this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] In this embodiment, an example will be described in which a computer device operates as an image processing device, and generates an image of the target (generated image) using an image generation AI (artificial intelligence) with input of coordinates of joint positions (joint information) of the target, and determines the consistency between the input joint information and the generated image. The target is a person or an animal.

[0013] <Hardware configuration> First, the hardware configuration of the image processing apparatus of this embodiment will be described with reference to FIG.

[0014] FIG. 1(a) is a block diagram showing the hardware configuration of an image processing device 1 according to this embodiment.

[0015] In this embodiment, a computer device operates as the image processing device 1. The processing of the image processing device of this embodiment may be realized by a single computer device, or may be realized by distributing the functions among multiple computer devices as necessary. The multiple computer devices are connected to each other so that they can communicate with each other.

[0016] The image processing device 1 includes a control unit 11, a nonvolatile memory 12, a work memory 13, a storage device 14, an input device 15, an output device 16, a communication interface 17, and a system bus 18.

[0017] The control unit 11 has a processor (CPU) that performs arithmetic processing and control processing of the image processing device 1. The non-volatile memory 12 is a ROM that stores programs and parameters executed by the processor of the control unit 11. The work memory 13 is a RAM that temporarily stores programs and data supplied from external devices. The storage device 14 is an internal device such as a hard disk or memory card built into the image processing device 1, or an external device such as a hard disk or memory card detachably connected to the image processing device 1. The input device 15 is an operating member such as a mouse, keyboard, or touch panel that accepts user operations and outputs operation information to the control unit 11. The output device 16 is a display device such as a monitor or display device composed of an LCD or organic EL, and displays data held by the image processing device 1 or data supplied from external devices. The communication interface 17 is communicably connected to a network such as the Internet or a LAN (Local Area Network). The system bus 18 includes an address bus, a data bus, and a control bus that connect each of the components 11 to 17 of the image processing device 1 to exchange data.

[0018] The nonvolatile memory 12 stores an OS (operating system), which is basic software executed by the control unit 11, and applications that cooperate with the OS to realize applied functions. In this embodiment, the nonvolatile memory 12 also stores applications that enable the image processing device 1 to perform consistency determination processing and occlusion determination processing, which will be described later.

[0019] The processing of the image processing device 1 of this embodiment is realized by loading software provided by an application. Note that the application has software for utilizing the basic functions of the OS installed in the image processing device 1. Note that the OS of the image processing device 1 may have software for realizing the processing of this embodiment.

[0020] <Functional configuration> Next, functional blocks of the image processing device of this embodiment will be described with reference to FIG.

[0021] FIG. 1(b) is a functional block diagram of the image processing device 1 of this embodiment.

[0022] The image processing device 1 includes a joint information input unit 101, an image generation unit 102, a joint detection unit 103, an occlusion determination unit 104, and a consistency determination unit 105. Each function of the image processing device 1 is configured using hardware and software. Note that each functional unit may be configured as a system consisting of one or more computer devices or server devices connected via a network.

[0023] The joint information input unit 101 inputs joint information to set the posture of an image (generated image) generated by the image generation unit 102. The generated image is an image of the object to be generated itself or an image including the object to be generated. In this embodiment, the generated image is an image including a part of the human body (upper body, lower body, left body, right body, etc.) or the whole body.

[0024] The joint information is used to set the posture of the object to be generated by the image generation unit 102. The joint information includes information such as an object ID that identifies the object to be generated, a joint ID that identifies the location of the joint for each object to be generated, and coordinates (x, y) of the joint in the image for each object to be generated.

[0025] The image generation unit 102 has an image generator realized by a known diffusion model or the like, and generates an image based on the joint information acquired from the joint information input unit 101 .

[0026] The joint detection unit 103 infers the joint positions of the target joints included in the image generated by the image generation unit 102 by deep learning, which is one type of machine learning, and generates joint detection results.

[0027] The occlusion determination unit 104 determines the occlusion state of each joint based on the joint information (input joint information) acquired from the joint information input unit 101, and generates occlusion information. The occlusion determination method will be described in detail later.

[0028] The consistency determination unit 105 determines the consistency between the input joint information and the generated image based on the input joint information acquired from the joint information input unit 101, the joint detection result of the generated image acquired from the joint detection unit 103, and the occlusion information acquired from the occlusion determination unit 104. Details of the consistency determination method will be described later.

[0029] A possible application example of this embodiment is generating images based on joint information and using the generated images as training data for machine learning. To achieve robust learning in machine learning, a large amount of training data is required, but generating a large amount of training data is not easy. In this case, a large amount of training data can be generated by generating images based on joint information, but the generated images contain images that are inconsistent with input information that is inappropriate for learning. Therefore, in this embodiment, inconsistent images contained in the generated images are identified and excluded from the training data, making it possible to generate machine learning training data for various tasks.

[0030] In this embodiment, the learning process is executed by the control unit 11, but this is not limiting. The image processing device 1 may also be equipped with a GPU (Graphics Processing Unit), and various types of arithmetic processes may be performed by the GPU. The GPU is an arithmetic processor that performs parallel arithmetic processing of data. The GPU is useful when performing multiple learning processes using neural networks, such as deep learning, or when performing multiple product-sum operations in inference processing. While an LSI is used for the GPU, equivalent functions may also be realized by a reconfigurable logic circuit known as an FPGA.

[0031] <Background> Next, a problem anticipated by this embodiment will be described with reference to FIG.

[0032] FIG. 2 shows an example of input joint information and a generated image when generating an image in this embodiment.

[0033] Figures 2(a) and 2(e) show examples of input joint information. Images are generated by inputting such input joint information into an image generator. Figures 2(b) to 2(d) show examples of images generated based on the input joint information of Figure 2(a). Figures 2(f) and 2(g) show examples of images generated based on the input joint information of Figure 2(e).

[0034] In the examples of Figures 2(a) to (d), it is desirable to generate an image like that shown in Figure 2(b) for the input joint information in Figure 2(a). However, as an example of a generated image, an image may be generated in which the right arm, indicated by the dashed line, is not present, as shown in Figure 2(c). This is called an absence. Also, as shown in Figure 2(d), an image may be generated in which an arm, indicated by the diagonal line, has been added. This is called an addition.

[0035] In the examples of Figures 2(e) to 2(g), it is desirable to generate an image in which the right arm is hidden, as shown in Figure 2(f), in response to input joint information in which the right arm is hidden in Figure 2(e). However, as an example of a generated image, as shown in Figure 2(g), the right arm shown in the grid may be generated in a location different from its original position.

[0036] Inconsistent generated images such as those in Figures 2(c), 2(d), and 2(g) may hinder correct learning when used as training data for machine learning, so it is necessary to generate training data from which these inconsistent images have been removed.

[0037] However, when an image like that shown in Figure 2(f) is generated for input joint information like that shown in Figure 2(e), it is difficult to determine whether the right arm is hidden or missing. This is because the posture of the object to be generated changes depending on the input joint information, so it is difficult to determine whether the right arm should be treated as occluded or missing based on the input joint information alone.

[0038] Therefore, in this embodiment, a joint occlusion determination is performed based on input joint information, and consistency determination is performed on the input joint information, the occlusion determination result, and the joint detection result of the generated image to determine whether there are additions or omissions in the generated image.The consistency determination result is then used to remove generated images in which additions or omissions have occurred, making it possible to generate high-quality training data.

[0039] In this embodiment, the generation of learning data has been exemplified as an application example of determining the consistency of a generated image, but the present invention is not limited to this and may be applied to an online service in which a user inputs joint information to generate an image, etc.

[0040] <Consistency determination method> Next, the consistency determination method of this embodiment will be described with reference to FIG.

[0041] The joint information input unit 101 inputs the joint information to the image generation unit 102 and the consistency determination unit 105 .

[0042] Joint information is used to specify the posture of the target to be generated during image generation. Table 1 shows an example of the data structure of joint information. In Table 1, the joint information includes a target ID that identifies the target to be generated in the image, a joint ID that identifies the joint location for each target ID, and x and y coordinates that indicate the position of the joint for each target ID in the image. For example, the data in the first row of Table 1 indicates that the target ID is 1, which is joint information for the person with target ID 1. A joint ID of 1 indicates the neck, for example. The parts associated with the joint IDs are predefined, such as the neck for a part with joint ID 1 and the top of the head for a part with joint ID 2. The x coordinate is 212 and the y coordinate is 540, indicating that the part (neck) with joint ID 1 of the person with target ID 1 is located at the x coordinate 212 and the y coordinate 540 in the image. The joint information table in Table 1 records multiple rows of joint information data.

[0043] Furthermore, this embodiment is based on the premise that all joint information for a target for which a target ID is to be generated is available. For example, this is the case when a person's joint information is generated using CG (Computer Graphics). Joint information can also be manually added to an image that has actually been captured (a live-action image). In this case, it is difficult to add joint information to a hidden part, and the corresponding joint information is missing. In such a case, as will be described later, the missing joint information may be processed by determining that the joint is an occluded joint. (Table 1) TIFF2025161527000002.tif44162 Next, the image generation unit 102 generates an image based on the input joint information. The image generation unit 102 includes an image generator implemented using known techniques such as a diffusion model, GAN (Generative Adversarial Networks), or VAE (Variational Auto-Encoder). The image generation unit 102 inputs the joint information acquired by the joint information input unit 101 into the image generator and generates an image based on the posture specified by the joint information. The number of dimensions of the input joint information is expanded or reduced to match the format of the image generator. Furthermore, in addition to the joint information, text or depth information may be input to the image generator, or the image generator may be switched depending on the sport or scene to be generated, allowing for detailed specification of the sport or scene to be generated.

[0044] Next, the joint detection unit 103 infers the joint positions of the generated image (target) and generates a joint detection result. The inference process of the joint detection unit 103 is realized by a known technology such as machine learning. Examples include OpenPose and DeepPose. The joint detection result generated by the joint detection unit 103 includes, like the input joint information shown in Table 1, a target ID that identifies the joint of the target in the image, a joint ID that identifies the location of the joint for each target ID, and x and y coordinates that indicate the position of the joint for each target ID in the image.

[0045] Next, the occlusion determination unit 104 determines the occlusion state for each joint in the input joint information and generates occlusion information. A state in which a joint is occluded means that the joint to be generated or a large portion of the part to which the joint belongs is hidden, and includes occlusion by other parts of the human body or occlusion by other objects. The occlusion determination is performed based on the joint information acquired from the joint information input unit 101. The occlusion determination method based on the input joint information or the input joint information and the generated image will be described later.

[0046] Next, the consistency determination unit 105 determines the consistency of the generated image based on the input joint information acquired from the joint information input unit 101, the occlusion information acquired from the occlusion determination unit 104, and the joint detection result of the generated image acquired from the joint detection unit 103.

[0047] FIG. 3 is a flowchart illustrating a process in which the consistency determining unit 105 determines the consistency of the generated image.

[0048] The processing in Fig. 3 is realized by the control unit 11 executing a program stored in the nonvolatile memory 12 and functioning as each block in Fig. 1(b). The processing in Fig. 3 is executed for all generation targets and all joints included in the generated image.

[0049] In step S301, the consistency determination unit 105 acquires input joint information from the joint information input unit 101, occlusion information from the occlusion determination unit 104, and joint detection results from the joint detection unit 103.

[0050] In step S302, the consistency determination unit 105 sequentially acquires, as the joint information of the determination target, the joint information of the target ID from the input joint information.

[0051] In step S303, the consistency determination unit 105 determines whether the joint to be determined is occluded by referring to the occlusion information of the joint to be determined. If the consistency determination unit 105 determines that the joint to be determined is occluded, the process proceeds to step S304, and if the consistency determination unit 105 determines that the joint to be determined is not occluded, the process proceeds to step S305.

[0052] In step S304, since it has been determined in step S303 that the joint is occluded, it can be determined that there is consistency if there is no joint detection result corresponding to the input joint information of the determination target. For example, if it is determined that the neck, which is joint ID 1 of the person with target ID 1 in the first row of Table 1, is occluded, it is determined that there is consistency if there is no joint detection result for joint ID 1 of the person with target ID 1. Therefore, the determination in step S304 is made based on whether there is a joint detection result corresponding to the target ID and joint ID of the input joint information of the determination target.

[0053] In step S304, if there is a joint detection result corresponding to the input joint information to be determined, the consistency determination unit 105 proceeds to step S306 and determines that there is no consistency. On the other hand, if there is no joint detection result corresponding to the input joint information to be determined, the consistency determination unit 105 proceeds to step S307 and determines that there is consistency.

[0054] In the examples of Figures 2(e) and 2(g), the joints of the right arm, such as the right wrist and right elbow, are occluded. Therefore, if an image like Figure 2(f) is generated, the joints corresponding to the right wrist and right elbow are not detected, and it is determined that there is consistency. On the other hand, if an image like Figure 2(g) is generated, the joints corresponding to the right wrist and right elbow are detected, and it is determined that there is no consistency.

[0055] In step S305, since it has been determined in step S303 that the joint is not occluded, it can be determined that there is consistency if there is a joint detection result corresponding to the input joint information of the determination target. For example, if it is determined that the neck, which is joint ID 1 of the person with target ID 1 in the first row of Table 1, is not occluded, it can be determined that there is consistency if the joint with joint ID 1 of the person with target ID 1 is detected near x coordinate 212 and y coordinate 540. In this case, for example, it can be determined using Equation 1 whether the joint position of the input joint information and the joint position of the corresponding joint detection result are within a range that does not exceed a threshold. (Formula 1) √([(x_d-x_i)]^2+[(y_d-y_i)]^2) <r_th In Equation 1, ^2 represents a power of 2, xi and yi are the x and y coordinates of the joint to be determined in the input joint information, and xd and yd are the x and y coordinates of the joint to be determined in the joint detection result. Furthermore, r_th is an arbitrarily set threshold value, and determines whether the joint positions xd and yd of the joint to be determined in the joint detection result are within the range of a circle with r_th as the radius for the joint positions xi and yi of the joint to be determined in the input joint information. The match of part classification is determined by determining whether the joint ID of the joint to be determined included in the input joint information matches the joint ID of the joint to be determined obtained by the joint detection unit 103.

[0056] In step S305, if there is a joint detection result corresponding to the input joint information to be determined, the consistency determination unit 105 advances the process to step S308 and determines that there is consistency. On the other hand, if there is no joint detection result corresponding to the input joint information to be determined, the consistency determination unit 105 advances the process to step S309 and determines that there is no consistency. In this case, if there is no joint detection result corresponding to the input joint information, it means that there is no joint. On the other hand, if there is a joint detection result corresponding to the input joint information but it does not satisfy the condition of Equation 1, it is possible that the part exists but has a different posture from the input joint information. On the other hand, if there are two or more joint detection results corresponding to the input joint information, it is possible that a joint has been added. If there is an addition, a determination is made in step S311, which will be described later.

[0057] In the examples of FIGS. 2(a) to 2(d), joints such as the right wrist and right elbow of the right arm are not occluded, so it is desirable to generate an image like that of FIG. 2(b). If the right wrist and right elbow of the input joint information in FIG. 2(a) match the right wrist and right elbow detected from the generated image in FIG. 2(b), it can be determined that there is consistency. On the other hand, if the right wrist and right elbow joints are missing as in FIG. 2(c), the right wrist and right elbow joints cannot be detected, and therefore it is determined that there is no consistency. Furthermore, if a right hand is added as in FIG. 2(d), there will be two or more joints corresponding to the right wrist and right elbow of the input joint information, and therefore it is determined that there is no consistency. The determination of such addition will be explained in step S311, which will be described later.

[0058] In step S310, the consistency determination unit 105 determines whether consistency determination has been completed for all joints included in the input joint information for a certain target ID. If consistency determination has been completed, the consistency determination unit 105 proceeds to step S311. If consistency determination has not been completed, the consistency determination unit 105 returns to step S302 and repeats the process until determination of all joint IDs for a certain target ID has been completed.

[0059] In step S311, the consistency determination unit 105 determines whether or not a corresponding consistency determination result exists for all joint detection results associated with a certain target ID, and ends the process if a consistency determination result exists for all joint detection results. If a consistency determination result does not exist for all joint detection results, the consistency determination unit 105 proceeds to step S312.

[0060] In step S312, the consistency determination unit 105 determines that a part has been added to the generated image, such as the generated image in which an arm not specified in the input joint information in Figure 2(d) has been added, and determines that there is no consistency.

[0061] In this embodiment, the joint information of a person with a certain target ID is assumed to be complete, but for example, when manually adding joint information to a live-action image, it is difficult to add joint information to hidden parts, and such joint information is missing. In such cases, the absence of input joint information may be treated as an occluded joint, and the same processing as above may be performed.

[0062] By performing the consistency determination as described above, it is possible to identify images from a plurality of generated images that contain events such as the addition or absence of a joint or a different position.

[0063] <Method for automatically generating input joint information> Next, the process of automatically generating input joint information will be described with reference to FIGS.

[0064] When generating a large number of images based on joint information and using them as learning data for machine learning, as in the application example of this embodiment, it is necessary to prepare a large number of different types of joint information. However, since manually generating a large amount of joint information for multiple people is labor-intensive, it is desirable to automatically generate joint information. For example, in volleyball, when a player attacks, there are players blocking nearby and players in receiving positions behind them. In this way, the posture of a person is related to the postures of the main subject and people around him / her. Therefore, it is required that the postures of each person included in the automatically generated joint information are related to each other. Therefore, below, an example is described in which the postures of other surrounding objects to be generated from the posture of a certain object to be generated by learning the correlation between the postures of the objects to be generated in the sport or scene to be generated, and related joint information is automatically generated.

[0065] FIG. 4 illustrates an example of a functional block in which a joint information generating unit is added to the configuration of FIG. 1(b).

[0066] The joint information generation unit 106 receives input of joint information that serves as a model for the sport or scene to be generated, such as when joint information is manually input for a live-action image or when joint information is extracted from a live-action image using a joint detection model, and information representing the target to be generated that corresponds to the main subject among the joint information.The joint information generation unit 106 then generates a joint information generation model by learning the correlation between the joint information of the main subject and the joint information of surrounding objects.Here, correlation includes the position of each joint of the target to be generated and the relative position, size, orientation, etc. of other objects.The joint information generation model is realized using machine learning or a lookup table.

[0067] The joint information generation unit 106 inputs manually input, manually selected, or automatically selected joint information of the main subject into the trained joint information generation model, and generates joint information of peripheral objects related to the target to be generated. After that, when generating other objects, the joint information of the main subject and peripheral objects may be used.

[0068] 5 illustrates an example of joint information generated by the joint information generation unit 106. When first joint information 501, which is the main subject performing an attack in volleyball, is determined and input to the joint information generation model, second joint information 502 performing a block in relation to the joint information is generated. After that, joint information 503 performing a block is further generated in relation to the joint information 501 performing an attack and the joint information 502 performing a block. Note that the joint information generation model may be switched depending on the purpose, such as for each sport or scene.

[0069] Furthermore, instead of automatically generating input joint information using a joint information generation model, joint information may be generated by the following method without using a joint information generation model.

[0070] The joint information generation unit 106 generates joint information based on motion information that records changes in joint position accompanying the motion of the object to be generated, and behavior pattern information that records behavioral conditions related to the motion of the object to be generated. For example, the motion information of the main subject is a running motion and a shooting motion in soccer, and the behavior pattern information is movement near the goal. Furthermore, the motion information of other people around the main subject is a running motion and a sliding motion in soccer, and the behavior pattern information is movement near the main subject. Using this motion information and behavior pattern information, a simulation is performed and joint information related to the motions of the main subject and other people around it is output.

[0071] In this way, by generating the postures of other surrounding objects from the posture of a certain object to be generated and automatically generating related joint information, it is possible to efficiently generate a variety of images in which the postures of other objects are related to each other.

[0072] <How to generate training data> Next, generation of learning data will be described with reference to FIG.

[0073] When using generated images as training data, which is one application example of this embodiment, inconsistent generated images may hinder correct learning in machine learning. Therefore, based on the consistency determination result, inconsistent generated images are excluded and appropriate training data is generated.

[0074] FIG. 6 illustrates a configuration in which an image selection unit is added to the configuration of FIG. 1(b).

[0075] The image selection unit 107 selects a generated image that has consistency based on the determination result of the consistency determination unit 105. This makes it possible to collect only consistent data. Note that it is also possible to collect only inconsistent data. The image selection unit 107 may then reflect information required as learning data in the joint information, such as reflecting the joint detection result by the joint detection unit 103 in the joint information, reflecting the occlusion determination result by the occlusion determination unit 104 in the joint information, reflecting the occlusion rate of the joint calculated by the occlusion determination unit 104 in the joint information, or reflecting the consistency determination result by the consistency determination unit 105 in the joint information.

[0076] In this way, by selecting consistent generated images based on the consistency determination results, it is possible to generate training data consisting only of consistent images.

[0077] <Occupation judgment method> Next, the occlusion determination method will be described with reference to FIGS.

[0078] The occlusion determination unit 104 performs occlusion determination based on the input joint information acquired from the joint information input unit 101. The occlusion determination unit 104 may also perform occlusion determination based on the generated image generated by the image generation unit 102 in addition to the input joint information.

[0079] Fig. 7 illustrates a functional block diagram for performing occlusion determination based on input joint information and a generated image. Fig. 8 is a flowchart illustrating occlusion determination processing based on input joint information and a generated image. Note that Fig. 8 includes occlusion determination processing based only on input joint information.

[0080] The process of making an occlusion determination based on input joint information and the process of making an occlusion determination based on input joint information and a generated image will be described below with reference to FIG.

[0081] In this embodiment, occlusion determination is performed using a three-dimensional surface model, which allows for accurate occlusion determination and also takes into account the ambiguity of occlusion based on the occlusion rate of the part region.

[0082] In step S801, the occlusion determination unit 104 determines whether the joint coordinates included in the input joint information are three-dimensional coordinates. If the occlusion determination unit 104 determines that the joint coordinates included in the input joint information are three-dimensional coordinates, the process proceeds to step S803. If the occlusion determination unit 104 determines that the joint coordinates included in the input joint information are not three-dimensional coordinates, the process proceeds to step S802.

[0083] In step S802, if the joint coordinates are two-dimensional coordinates, the occlusion determination unit 104 expands the joint coordinates to three-dimensional coordinates. The expansion method can be realized by existing technology, for example, by using machine learning that learns the relationship between the three-dimensional coordinates of the object to be generated and the coordinates projected onto a two-dimensional plane. By using such a machine learning model, it is possible to restore three-dimensional coordinates using the coordinates projected onto a two-dimensional plane as input.

[0084] In step S803, the occlusion determination unit 104 applies the input joint information to the three-dimensional surface model to match the posture of the generated image (target) with that of the three-dimensional surface model. The three-dimensional surface model is a general-purpose model that has joint positions corresponding to the input joint information and the parts and regions to which each joint belongs, and is based on a model consisting of only the basic parts that represent the target. For example, in the case of a human body, the base would be a three-dimensional surface model that has the head, neck, chest, abdomen, waist, upper arms, forearms, upper thighs, lower legs, etc.

[0085] Here, a method for matching the posture of a generated image (object to be generated) with that of a three-dimensional surface model will be described with reference to Fig. 9. In Fig. 9, an image 902 is an image generated by inputting input joint information 901 into an image generation model. By applying the input joint information 901 to a three-dimensional surface model 903 having a basic structure of a previously prepared object to be generated and changing the joint positions, a three-dimensional surface model 904 with a changed posture is generated. This makes it possible to match the posture of the generated image 902 with that of the three-dimensional surface model 904.

[0086] In step S804, the occlusion determination unit 104 determines whether a generated image has been input, and if it is determined that a generated image has been input, the process proceeds to step S805, where the body shape of the base three-dimensional surface model is deformed to match the generated image, thereby generating a deformed three-dimensional surface model. If the occlusion determination unit 104 determines that a generated image has not been input, the process proceeds to step S808.

[0087] In step S808, the occlusion determination unit 104 performs occlusion determination using the underlying three-dimensional surface model.

[0088] In step S805, the occlusion determination unit 104 determines whether or not it is necessary to change the elements constituting the three-dimensional surface model in accordance with scene information such as the sport or scene of the generated image (target image). For example, some sports, such as American football and kendo, require the wearer to wear protective gear. In such sports, the degree of occlusion varies depending on the protective gear, so occlusion can be determined more accurately by adding elements corresponding to the protective gear to the elements constituting the three-dimensional surface model. The scene information is estimated from the generated image by the image generation unit 102, or is obtained in advance from the joint information input unit 101. If the occlusion determination unit 104 determines that it is necessary to change the elements, the process proceeds to step S806. If it determines that it is not necessary to change the elements, the process skips step S806 and proceeds to step S807.

[0089] In step S806, the occlusion determination unit 104 changes the components of the three-dimensional surface model according to the scene information. Scene information and the components of the corresponding three-dimensional planar model are defined in advance. For example, if the scene is American football or kendo, parts corresponding to a helmet or protectors are added to the head, shoulders, and chest of the three-dimensional surface model.

[0090] In step S807, the occlusion determination unit 104 deforms the size of each part of the three-dimensional surface model so that the body shape of the generated image (target) matches that of the three-dimensional surface model. The deformation of the parts is performed by enlarging or reducing the area of ​​each part region on the projection surface of the three-dimensional surface model so that the area of ​​the part region in the generated image (target) matches that of the three-dimensional surface model. Furthermore, the deformation of the part changed in step S806 is performed in conjunction with the deformation of the corresponding part. The part region of the generated image (target) is detected by performing segmentation on the generated image for each part. Segmentation is achieved using known technology, for example, by dividing the part region of the generated image (target) based on information about each part of the human body that is prepared in advance. In this case, it is desirable that the physical characteristics of the generated image (target) have been learned in advance. Note that if a joint is missing in the input joint information, the corresponding part is excluded from the deformation.

[0091] In step S808, the occlusion determination unit 104 performs occlusion determination using the three-dimensional surface model generated in steps S806 and S807. The occlusion determination is realized by a known technique, such as ray casting. In this case, a ray is projected onto the three-dimensional surface model placed in three-dimensional space, starting from the camera position, at the position of the joint to be occluded. If the ray intersects with an area other than the part area to which the joint to be occluded belongs, occlusion is determined. In this case, the occlusion rate of the part area is calculated from the displayed area relative to the total area of ​​the part area to be determined on the projection plane. If the occlusion rate is equal to or less than a threshold, occlusion is uncertain, and the consistency determination unit 105 may exclude the part area from the consistency determination target.

[0092] In addition, there are cases where the input joint information is missing joint information, such as record 20 in Table 2. For example, when joint information is manually added to a real-life image, it is difficult to add joint information to a hidden part, and the corresponding joint information is missing. In this case, the joint with missing joint information is determined to be occluded, and the part region of the 3D surface model to which the joint belongs is deleted before performing the occlusion determination, or the part region is treated as an area not involved in the occlusion determination. (Table 2) TIFF2025161527000003.tif53170By using a three-dimensional surface model in this way, accurate occlusion determination can be performed, and the occlusion rate of the part area to be determined as the occlusion target can be calculated, allowing consistency determination to be performed while excluding joints whose occlusion is unclear.

[0093] As described above, according to this embodiment, it is possible to determine the consistency between the joint information input at the time of image generation and the generated image. Furthermore, when a specific part is missing or added in the generated image, it is possible to determine whether this is a normal state caused by occlusion or an abnormal state caused by the absence during image generation.

[0094] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0095] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0096] The disclosure of this specification includes the following image processing device, image processing method, and program. [Configuration 1] an input means for inputting joint information to be generated; a generation means for generating an image of the object to be generated based on the joint information; a detection means for detecting joints from the generated image generated by the generation means; an occlusion determination means for determining an occlusion state of a joint in the generated image; and a consistency determination unit that determines consistency between the joint information and the generated image based on the joint detection result and an occlusion state of the joint. [Configuration 2] the occlusion determination means determines the occlusion state based on the joint information; the consistency determination means determines that there is no consistency when the occlusion determination means determines that a joint is occluded and the detection means detects the joint; 2. The image processing device according to configuration 1, wherein the image processing device determines that the joint is occluded by the occlusion determination means and that the joint is not detected by the detection means. [Configuration 3] the occlusion determination means determines the occlusion state based on the joint information; the consistency determination means determines that the consistency exists when the occlusion determination means determines that the joint is not occluded and the detection means detects the joint; 3. The image processing device according to configuration 1 or 2, wherein the image processing device determines that there is no consistency when the occlusion determination means determines that a joint is occluded and the detection means detects the joint. [Configuration 4] 4. The image processing device according to any one of configurations 1 to 3, wherein the consistency determination means determines the consistency for a plurality of pieces of joint information input by the input means based on detection results of the plurality of joints and occlusion states of the plurality of joints. [Configuration 5] 5. The image processing device according to any one of configurations 2 to 4, wherein, when a plurality of joints are detected from the generated image by the detection means, the occlusion determination means determines that there is no consistency if there is no consistency determination result for the detection results of the plurality of joints. [Configuration 6] a joint information generating means for generating the joint information, 6. The image processing device according to any one of configurations 1 to 5, wherein the joint information generation means generates the joint information by performing a learning process using a trained model, using manually or automatically generated joint information as an input. [Configuration 7] The image processing device according to configuration 6, wherein the joint information generating means generates the joint information based on motion information that records changes in joint positions accompanying the movement of the object to be generated, and action pattern information related to the movement of the object to be generated. [Configuration 8] a selection means for selecting the generated image having the consistency according to the consistency determination means; 8. The image processing device according to any one of configurations 1 to 7, wherein the consistent generated image is used in a learning process. [Configuration 9] The image processing device according to configuration 8, wherein the selection means reflects the joint detection result for the generated image, the occlusion state of the joint, the occlusion rate of the joint for the generated image, and the consistency determination result in the joint information. [Configuration 10] the occlusion determination means determines the occlusion state based on the joint information and the generated image; the consistency determination means determines that there is no consistency when the occlusion determination means determines that a joint is occluded and the detection means detects the joint; 2. The image processing device according to configuration 1, wherein the image processing device determines that the joint is occluded by the occlusion determination means and that the joint is not detected by the detection means. [Configuration 11] the occlusion determination means determines the occlusion state based on the joint information and the generated image; the consistency determination means determines that the consistency exists when the occlusion determination means determines that the joint is not occluded and the detection means detects the joint; 3. The image processing device according to configuration 1 or 2, wherein the image processing device determines that there is no consistency when the occlusion determination means determines that a joint is occluded and the detection means detects the joint. [Configuration 12] the occlusion determination means determines an occlusion state using a three-dimensional surface model having the same orientation as the generated image; 12. The image processing device according to any one of configurations 1 to 11, wherein the three-dimensional surface model is generated based on the joint information. [Configuration 13] 13. The image processing device according to configuration 12, wherein the three-dimensional surface model is deformed to match the body shape of the generated image. [Configuration 14] 14. The image processing device according to claim 12, wherein components of the three-dimensional surface model are changed based on scene information of the generated image. [Configuration 15] 15. The image processing device according to configuration 14, wherein the scene information is inferred from the generated image or obtained by the input means. [Configuration 16] 16. The image processing device according to any one of configurations 1 to 15, wherein the occlusion determination means excludes the joint from the consistency determination target when the occlusion rate of the joint in the generated image is equal to or less than a threshold value. [Configuration 17] 17. The image processing device according to any one of configurations 1 to 16, wherein the joint information and the joint detection results include information for identifying the object to be generated, information for identifying the location of the joint of the object to be generated, and coordinates indicating the position of the joint of the object to be generated. [Configuration 18] an input step of inputting joint information to be generated by an input means; A generating step in which a generating means generates an image of the object to be generated based on the joint information; a step of detecting joints from the generated image by a detection means; an occlusion determination means for determining an occlusion state of a joint in the generated image; a consistency determination means for determining consistency between the joint information and the generated image based on the joint detection result and the occlusion state of the joint. [Configuration 19] 18. A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 17. [Explanation of symbols]

[0097] 1... image processing device, 11... control unit, 101... joint information input unit, 102... image generation unit, 103... joint detection unit, 104... occlusion determination unit, 105... consistency determination unit

Claims

1. an input means for inputting joint information to be generated; a generation means for generating an image of the object to be generated based on the joint information; a detection means for detecting joints from the generated image generated by the generation means; an occlusion determination means for determining an occlusion state of a joint in the generated image; and a consistency determination unit that determines consistency between the joint information and the generated image based on the joint detection result and an occlusion state of the joint.

2. the occlusion determination means determines the occlusion state based on the joint information; the consistency determination means determines that there is no consistency when the occlusion determination means determines that a joint is occluded and the detection means detects the joint; 2. The image processing apparatus according to claim 1, wherein the consistency is determined to exist when the occlusion determination means determines that a joint is occluded and the detection means does not detect the joint.

3. the occlusion determination means determines the occlusion state based on the joint information; the consistency determination means determines that the consistency exists when the occlusion determination means determines that the joint is not occluded and the detection means detects the joint; 2. The image processing apparatus according to claim 1, wherein the inconsistency is determined when the occlusion determination means determines that a joint is occluded and the detection means detects the joint.

4. 2. The image processing device according to claim 1, wherein the consistency determination means determines the consistency of the plurality of pieces of joint information input by the input means based on detection results of the plurality of joints and occlusion states of the plurality of joints.

5. 3. The image processing device according to claim 2, wherein, when the detection means detects a plurality of joints from the generated image, the occlusion determination means determines that there is no consistency if there is no consistency determination result for the detection results of the plurality of joints.

6. a joint information generating means for generating the joint information, The image processing device according to claim 1 , wherein the joint information generating means generates the joint information by performing a learning process using a trained model, using manually or automatically generated joint information as an input.

7. 7. The image processing device according to claim 6, wherein the joint information generating means generates the joint information based on motion information that records changes in joint positions accompanying the movement of the object to be generated, and action pattern information related to the movement of the object to be generated.

8. a selection means for selecting the generated image having the consistency according to the consistency determination means; The image processing device according to claim 1 , wherein the consistent generated image is used for learning processing.

9. The image processing device according to claim 8, wherein the selection means reflects the joint detection result for the generated image, the occlusion state of the joint, the occlusion rate of the joint for the generated image, and the consistency determination result in the joint information.

10. the occlusion determination means determines the occlusion state based on the joint information and the generated image; the consistency determination means determines that there is no consistency when the occlusion determination means determines that a joint is occluded and the detection means detects the joint; 2. The image processing apparatus according to claim 1, wherein the consistency is determined to exist when the occlusion determination means determines that a joint is occluded and the detection means does not detect the joint.

11. the occlusion determination means determines the occlusion state based on the joint information and the generated image; the consistency determination means determines that the consistency exists when the occlusion determination means determines that the joint is not occluded and the detection means detects the joint; 2. The image processing apparatus according to claim 1, wherein the inconsistency is determined when the occlusion determination means determines that a joint is occluded and the detection means detects the joint.

12. the occlusion determination means determines an occlusion state using a three-dimensional surface model having the same orientation as the generated image; The image processing apparatus according to claim 1 , wherein the three-dimensional surface model is generated based on the joint information.

13. The image processing apparatus according to claim 12, wherein the three-dimensional surface model is deformed to match the body shape of the generated image.

14. The image processing apparatus according to claim 12 , wherein components of the three-dimensional surface model are changed based on scene information of the generated image.

15. 15. The image processing device according to claim 14, wherein the scene information is inferred from the generated image or obtained by the input means.

16. The image processing apparatus according to claim 1 , wherein the occlusion determination means excludes the joint from the consistency determination target when the occlusion rate of the joint in the generated image is equal to or less than a threshold value.

17. The image processing device according to claim 1, characterized in that the joint information and the joint detection results include information identifying the object to be generated, information identifying the location of the joint of the object to be generated, and coordinates indicating the position of the joint of the object to be generated.

18. an input step of inputting joint information to be generated by an input means; a generating step of generating an image of the object to be generated based on the joint information; a step of detecting joints from the generated image by a detection means; an occlusion determination means for determining an occlusion state of a joint in the generated image; a consistency determination means for determining consistency between the joint information and the generated image based on the joint detection result and the occlusion state of the joint.

19. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 17.

Images