Information processing device, information processing method, and program

Abstract

This generates a highly accurate model for extracting object representations. [Solution] The information processing device includes: a first generation means for extracting object representations of each object included in a target image and generating one or more first gaze regions based on said object representations; a second generation means for generating prompts based on the first gaze regions and generating one or more second gaze regions from the target image and the prompts using a pre-trained model; a loss calculation means for calculating a loss by referring to the first gaze regions and the second gaze regions; and a learning means for training the first generation means by referring to the loss.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Techniques for detecting an object from a target image are known. For example, Patent Document 1 discloses a learning apparatus that includes an object representation extraction unit and learns a model for extracting an object representation.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art as described above, since the model for extracting an object representation acquires the object representation by self-supervised learning, there is a problem that the accuracy of dividing the region to be focused on in the image of the object is low.

[0005] The present disclosure has been made in view of the above problems, and an exemplary object thereof is to provide a technique capable of efficiently learning a high-precision model for extracting an object.

Means for Solving the Problems

[0006] An information processing device relating to an illustrative aspect of this disclosure includes: a first generation means for extracting object representations of each object included in a target image and generating one or more first gaze regions based on said object representations; a second generation means for generating prompts based on the first gaze regions and generating one or more second gaze regions from the target image and the prompts using a pre-trained model; a loss calculation means for calculating a loss by referring to the first gaze regions and the second gaze regions; and a learning means for training the first generation means by referring to the loss. An example of an information processing method relating to this disclosure includes one or more processors extracting object representations of each object included in a target image using an extraction model, generating one or more first gaze regions based on said object representations, generating prompts based on the first gaze regions, generating one or more second gaze regions from the target image and the prompts using a pre-trained model, calculating a loss by referring to the first gaze regions and the second gaze regions, and training the extraction model by referring to the loss. An illustrative aspect of the present disclosure is an information processing program that causes a computer to function as an information processing device, wherein the computer functions as: a first generation means for extracting object representations of each object included in a target image and generating one or more first gaze regions based on said object representations; a second generation means for generating prompts based on the first gaze regions and generating one or more second gaze regions from the target image and the prompts using a pre-trained model; a loss calculation means for calculating a loss by referring to the first gaze regions and the second gaze regions; and a learning means for training the first generation means by referring to the loss. [Effects of the Invention]

[0007] One illustrative aspect of this disclosure demonstrates the effect of efficiently generating highly accurate models for extracting object representations through learning. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of the information processing device related to this disclosure. [Figure 2] This is a flowchart showing the flow of the information processing method related to this disclosure. [Figure 3] This is a block diagram showing the configuration of the information processing system related to this disclosure. [Figure 4] This diagram illustrates an example of processing in the information processing system related to this disclosure. [Figure 5] This diagram illustrates the processing flow in the information processing system related to this disclosure. [Figure 6] This diagram illustrates an example configuration and processing example of the information processing system related to this disclosure. [Figure 7] This diagram illustrates an example configuration and processing example of the information processing system related to this disclosure. [Figure 8] This diagram illustrates an example configuration and processing example of the information processing system related to this disclosure. [Figure 9] This is a block diagram showing the configuration of the information processing system related to this disclosure. [Figure 10] This is a block diagram showing the configuration of a computer that functions as an information processing device related to this disclosure. [Modes for carrying out the invention]

[0009] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining some or all of the technologies (things or methods) employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, the effects mentioned in each of the exemplary embodiments shown below are examples of effects that can be expected in that exemplary embodiment and do not define the scope of the present invention. That is, embodiments that do not produce the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

[0010] [First Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form for each of the exemplary embodiments described later. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur. Furthermore, each technology shown in the drawings referenced to explain this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur.

[0011] (Configuration of Information Processing Device 1) The configuration of the information processing device 1 according to this exemplary embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the information processing device 1. As shown in Figure 1, the information processing device 1 includes a first generation unit 11, a second generation unit 12, a loss calculation unit 13, and a learning unit 14.

[0012] (First generation unit 11) The first generation unit 11 acquires a target image, extracts the object representation of each object included in the target image, and generates one or more first fixation regions based on the object representation. Here, the object representation is a numerical representation of each object included in the target image, and as an example, it is represented as a vector. Also, the object representation is derived, for example, by referring to a feature map obtained from the target image. The first fixation region is information generated by referring to the object representation, and as an example, it is represented as one or more first fixation region maps. Note that the first generation unit 11, as an example, · An encoder to which the target image is input, · An extraction model that extracts the object representation of each object by referring to the output of the encoder can be configured to include, but this does not limit the exemplary embodiment.

[0013] (Second generation unit 12) The second generation unit 12 generates a prompt based on the first fixation region, and uses a pre-trained model to generate one or more second fixation regions from the target image and the prompt. Also, specific examples of the pre-trained model do not limit this exemplary embodiment, but as an example, SAM (Segment Anything Model) can be used.

[0014] (Loss calculation unit 13) The loss calculation unit 13 calculates a loss by referring to the first fixation region and the second fixation region. Specific examples of the loss function referred to by the loss calculation unit 13 do not limit this exemplary embodiment, but as an example, a loss function can be used that is configured such that the smaller the degree of difference between the first fixation region and the second fixation region, the smaller the value of the loss.

[0015] (Learning unit 14) The learning unit 14 trains the first generation unit 11 by referring to the loss calculated by the loss calculation unit 13. For example, the learning unit 14 updates the parameters of the extraction model used by the first generation unit 11 to extract the object representation of each object included in the target image by referring to the loss calculated by the loss calculation unit 13.

[0016] (Effects of Information Processing Device 1) As described above, in Information Processing Device 1, A first generation unit 11 extracts the object representation of each object contained in the target image and generates one or more first gaze regions based on said object representation, A second generation unit 12 generates a prompt based on the first gaze region and generates one or more second gaze regions from the target image and the prompt using a pre-trained model, A loss calculation unit 13 calculates the loss by referring to the first gaze region and the second gaze region, A learning unit 14 that teaches the first generation unit by referring to the loss, A configuration is adopted that includes the following. In this way, the information processing device 1 generates a first gaze region with the first generation unit 11, generates a second gaze region from the prompt and target image generated based on the first gaze region, calculates a loss by referring to these gaze regions, and trains the first generation unit 11 by referring to the calculated loss. Therefore, with the above configuration, a highly accurate model can be efficiently generated by accurately dividing the second gaze region and training a model for extracting object representations.

[0017] (Information processing method S1 flow) The flow of the information processing method S1 will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the information processing method S1. As shown in Figure 2, the information processing method S1 includes a step (process) S11 for generating a first gaze region, a step (process) S12 for generating a second gaze region, a step (process) S13 for calculating the loss, and a step (process) S14 for training the extraction model.

[0018] (Step S11) In step S11, the first generation unit 11 acquires a target image, extracts the object representation of each object contained in the target image using an extraction model, and generates one or more first gaze regions based on the object representations. A more detailed explanation of the first generation unit 11 has been given above, so it will be omitted here.

[0019] (Step S12) In step S12, the second generation unit 12 generates one or more second gaze regions from the target image and the prompt using a pre-trained model. A more detailed explanation of the second generation unit 12 has been given above, so it will be omitted here.

[0020] (Step S13) In step S13, the loss calculation unit 13 calculates the loss by referring to the first and second observation areas. A more detailed explanation of the loss calculation unit 13 has been given above, so it will be omitted here.

[0021] (Step S14) In step S14, the learning unit 14 trains the extraction model by referring to the loss. A more detailed explanation of the learning unit 14 has been given above, so it will be omitted here.

[0022] (Effects of Information Processing Method S1) As described above, in Information Processing Method S1, The object representation of each object contained in the target image is extracted using an extraction model, and one or more first gaze regions are generated based on these object representations. A prompt is generated based on the first gaze region, and one or more second gaze regions are generated from the target image and the prompt using a pre-trained model. The loss is calculated by referring to the first gaze region and the second gaze region. The extraction model is trained by referring to the loss. This configuration is employed. According to the above configuration, the same effect as that of the information processing device 1 is achieved.

[0023] [Second Embodiment] A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0024] (Configuration of Information Processing System 100A) The configuration of the information processing system 100A according to this exemplary embodiment will be described with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the information processing system 100A. As shown in Figure 3, the information processing system 100A includes an information processing device 1A and a monitoring device 60 and a group of cameras 70 connected to the information processing device 1A via a network N. Here, the specific configuration of the network N is not limited to this exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public telephone network, a mobile data communication network, or a combination of these networks can be used.

[0025] (Camera group 70) The camera group 70 comprises one or more cameras. For example, if the monitoring device 60 is configured as a traffic monitoring device, these cameras will capture images of roads, vehicles, etc. In another example, if the monitoring device 60 is configured as a security monitoring device inside a building, these cameras will capture images of people, vehicles, etc. inside or outside the building. The images captured by the camera group 70 are supplied to the information processing device 1A, for example.

[0026] (Monitoring device 60) In general terms, the monitoring device 60 acquires object representations and reconstructed images extracted by the information processing device 1A from images captured by the camera group 70, and performs monitoring processing on the target by referring to the acquired object representations and reconstructed images. As shown in Figure 3, the monitoring device 60 includes a control unit 61 and a communication unit 62. The communication unit 62 receives object representations and reconstructed images from the information processing device 1A. The control unit 61 then performs traffic monitoring processing by referring to the object representations and reconstructed images acquired from the information processing device 1A. As another example, the control unit 61 performs security monitoring processing within a building by referring to the object representations and reconstructed images acquired from the information processing device 1A.

[0027] In this exemplary embodiment, the monitoring device 60 is shown as a separate device from the information processing device 1A, but this does not limit this exemplary embodiment. The functions of the control unit 61 of the monitoring device 60 may also be provided by the control unit of the information processing device 1A.

[0028] (Configuration of Information Processing Device 1A) Next, the configuration of the information processing device 1A according to this exemplary embodiment will be described with reference to Figure 3. As shown in Figure 3, the information processing device 1A includes a control unit 10, a storage unit 20, a communication unit 30, and an input / output unit 40.

[0029] (Communications Section 30) The communication unit 30 communicates with devices outside the information processing device 1A. For example, the communication unit 30 communicates with the camera group 70. The communication unit 30 supplies images received from the camera group 70 to the control unit 10. The communication unit 30 also communicates with the monitoring device 60. The communication unit 30 supplies object representations and reconstructed images derived by the control unit 10 to the monitoring device 60.

[0030] (Input / output section 40) The input / output unit 40 is configured to include at least one of the following input / output devices: a keyboard, mouse, display, printer, touch panel, etc. Alternatively, the input / output unit 40 may be configured to have input / output devices such as a keyboard, mouse, display, printer, touch panel, etc. connected to it. In this configuration, the input / output unit 40 receives various types of information from the connected input device to the information processing device 1A. The input / output unit 40 also outputs various types of information to the connected output device under the control of the control unit 10. An interface such as USB (Universal Serial Bus) can be used as an example of the input / output unit 40.

[0031] (Storage unit 20) The storage unit 20 stores various data referenced by the control unit 10, as well as various data generated by the control unit 10. For example, the storage unit 20 stores: • Target image TI Prompt PT ·Object representation OR • First gaze region RI1 • Second gaze region RI2 Reconstructed image RE ·Reconstruction area RS ·Loss LS • Parameter group PG • Base model BM The following is stored. The target image TI may, for example, be an image supplied from the camera group 70, or it may not be. The target image TI contains, • In the learning phase, the images (training data) used for the learning process by the learning unit 14 described later are • Images (inference data) referenced by the object representation extraction unit 112 in the inference phase to extract object representations. It includes at least one of the following.

[0032] The prompt PT is supplementary information generated by the prompt generation unit 122 (described later) and input to the base model BM. Specific examples of the prompt PT will be described later.

[0033] The object representation OR is a representation extracted (generated) by the object representation extraction unit 112, described later, and is a numerical representation of each object included in the target image TI. For example, the object representation OR can be represented as a vector.

[0034] The first attention region RI1 is information generated by the object representation extraction unit 112 (described later) by referring to the object representation OR, and is represented, for example, as one or more first attention region maps. As will be described later, the object representation extraction unit 112 can be implemented, for example, using an attention model or a SLOT attention model. Correspondingly, the first attention region RI1 is sometimes referred to as an attention mask. The attention mask is a mask image that indicates which region of the target image TI is being focused on for representation extraction, and can be implemented, for example, as an alpha mask expressed using a multi-level opacity coefficient (also called an α coefficient). On the other hand, the second attention region RI2 is information generated by the second generation unit 12 (described later) by referring to the target image TI and using the base model BM, and is represented, for example, as one or more second attention region maps. Specific examples of the first attention region RI1 and the second attention region RI2 will be described later.

[0035] The reconstructed image RE is an image generated by the reconstruction unit 121, which will be described later, and is an image reconstructed by referring to the object representation of each object contained in the target image TI. The loss LS is calculated using a loss function by the loss calculation unit 13, which will be described later. Specific examples of the reconstructed image RE and loss LS will be described later.

[0036] The reconstructed region RS is information reconstructed from the object representation OR by the reconstruction unit 121, which will be described later, and is represented, for example, as one or more reconstructed region maps. The reconstruction unit 121 is sometimes referred to as the decoder 121. Correspondingly, the reconstructed region map is sometimes referred to as the decoder mask. The decoder mask is a mask image used in rendering (reconstruction processing) by the reconstruction unit (decoder) 121, and can be implemented, for example, as an alpha mask expressed using a multi-level opacity coefficient (also called the alpha coefficient). The decoder mask has a higher resolution and higher precision than the first attention region RI1 (attention mask) described above.

[0037] The parameter group PG includes parameters that define various models used by the control unit 10. For example, the parameter group PG includes: • One or more parameters that define the encoder used by the object representation extraction unit 112 - An extraction model used by the object representation extraction unit 112, which defines one or more parameters that define the extraction model for extracting object representations from the output of the encoder. • One or more parameters that define the decoder used by the reconstruction unit 121 These include, among others. At least some of these parameters are subject to learning (update) processing by the learning unit 14, which will be described later.

[0038] The base model BM is a pre-trained model used by the second generation unit 12, described later, to generate one or more second gaze regions RI2 from the target image TI. Specific examples of the base model BM are not limited to this exemplary embodiment, but one example is the use of a SAM (Segment Anything Model).

[0039] (Control Unit 10) As shown in Figure 3, the control unit 10 includes a first generation unit 11, a second generation unit 12, a loss calculation unit 13, a learning unit 14, and an output information generation unit 15.

[0040] (First generation unit 11) The first generation unit 11 acquires a target image TI, extracts object representations OR for each object contained in the target image TI, and generates one or more first gaze regions RI1 based on the object representations OR. As shown in Figure 3, the first generation unit 11 includes an acquisition unit 111 and an object representation extraction unit 112.

[0041] (Acquisition part 111) The acquisition unit 111 acquires the target data TI. The acquisition unit 111 may be configured to acquire the target image TI from the camera group 70 via the communication unit 30, or it may be configured to acquire the target image TI stored in the storage unit 20.

[0042] (Object expression extraction unit 112) The object representation extraction unit 112 extracts the object representation OR of each object included in the target image TI. As an example, the object representation extraction unit 112 extracts the object representation OR of each object included in the target image TI. • An encoder that receives a target image TI as input and generates a feature map of the target image TI, The system may also be configured to include an extraction model that extracts object representations OR of each object included in the target image TI by referring to the output (feature map) of the encoder. Here, a CNN (Convolutional Neural Network) may be used as the encoder. Furthermore, an attention model, more specifically a SLOT attention model, may be used as the extraction model. However, these examples do not limit the present exemplary embodiment. The object representation extraction unit 112 may also be configured to generate one or more first gaze regions RI1 by referring to the extracted object representations OR. More specifically, the object representation extraction unit 112 may be configured to generate one or more first gaze regions RI1 by referring to the extracted object representations OR. • By referring to one or more object representations OR, one or more mask images (attention masks) are generated as one or more first attention region maps. The attention mask represents the one or more first gaze regions RI1. This configuration is also acceptable.

[0043] (Second generation unit 12) The second generation unit 12 generates one or more second gaze regions RI2 from the target image TI and prompt PT using a pre-trained base model BM. For example, the second generation unit 12 may generate one or more second gaze region maps by referring to the target image TI and prompt PT, and the one or more second gaze regions RI2 may be represented by these second gaze region maps. The second gaze region (second gaze region map) can be realized as an alpha mask represented using a multi-level opacity coefficient (also called an α coefficient), for example. As shown in Figure 3, the second generation unit 12 includes a reconstruction unit 121 and a prompt generation unit 122.

[0044] (Reconstruction part 121) The reconstruction unit 121 reconstructs each of the objects by referring to the object representations of each object extracted by the object representation extraction unit 112. The reconstruction unit 121 generates a reconstructed image RE, for example, which includes the reconstructed objects. The reconstruction unit 121 may also be configured as a decoder that receives the object representations of each object as input and outputs the reconstructed image RE. Alternatively, the reconstruction unit 121 may be configured to generate one or more reconstructed regions RS by referring to the object representations OR. More specifically, the reconstruction unit 121, • A mask image (decoder mask) is generated as one or more reconstructed region maps by referencing one or more object representations OR. The decoder mask represents the one or more reconstruction regions RS. This configuration is also acceptable.

[0045] (Prompt generation unit 122) The prompt generation unit 122 is: • The first gaze region RI1 (attention mask) generated by the object representation extraction unit 112, and • Reconstruction region RS (reconstruction region map, decoder mask) generated by the reconstruction unit 121 A prompt PT is generated for input to the base model BM by referring to at least one of the following. This prompt is a prompt associated with the target image TI. This prompt PT may be a prompt in a predetermined format, or it may be a prompt having a format corresponding to the target image TI or input from the user. The prompt PT generated by the prompt generation unit 122 is input to the base model BM together with the target image TI and is referenced to generate one or more second gaze regions RI2.

[0046] To give a more specific example, the above prompt PT is: • Grid point prompt Language prompt • Rectangular prompt • Mask prompt Any of the following types of prompts may be used. Here, a grid point prompt is a prompt that specifies a target area (area of ​​interest) by one or more points, a language prompt is a prompt that specifies a target area by natural language, a rectangle prompt is a prompt that specifies a target area by a rectangle (or coordinate points to identify the rectangle), and a mask prompt is a prompt that specifies a target area by a mask (segmentation mask).

[0047] Furthermore, the prompt generation unit 122 may be configured to perform a sampling process on the input to the prompt generation unit 122 (attention mask or decoder mask) and then perform a prompt formatting process on the result of the sampling process. The sampling process can also be described as the process of extracting one or more points to be referenced in the prompt formatting process from the attention mask or decoder mask. Specific examples of the sampling process and prompt formatting process are not limited to this exemplary embodiment, but some examples are as follows.

[0048] (Specific example of sampling process) • Sampling process based on the mask score distribution, more specifically, sampling process that references the distribution of mask scores (alpha coefficients) of the attention mask or decoder mask. • A sampling process using a threshold on mask scores; more specifically, a process of extracting one or more points from the mask scores (alpha coefficients) of an attention mask or decoder mask that correspond to scores above a predetermined threshold. • Sampling process using the centroid of a mask; more specifically, a process that calculates the centroid of multiple masks contained in an attention mask or decoder mask, and extracts one or more points according to that centroid. • Sampling process using Gaussian sampling, more specifically, a sampling process that applies Gaussian sampling to an attention mask or decoder mask. • Sampling processes using negative sampling, more specifically, processes that use sampling (negative sampling) to specify non-object regions in an attention mask or decoder mask. • Sampling process with density control; more specifically, a sampling process that controls the density so that the density of points in the attention mask or decoder mask becomes sparse. • Exclusive sampling, more specifically, sampling in a way that each target region (each slot) indicated by the attention mask or decoder mask is mutually exclusive.

[0049] (Specific example of prompt formatting) • A process to generate a bounding box prompt; more specifically, a process to identify the bounding rectangle of multiple points obtained by the sampling process described above, and to represent the prompt using the identified bounding rectangle. • A process to generate a point prompt; more specifically, a process that uses one or more points obtained by the sampling process described above as the prompt. • A process to generate a mask prompt; more specifically, a process to generate a mask using one or more points obtained by the sampling process described above, and to represent the prompt using the generated mask. • A process for generating a language prompt, more specifically, a process for generating a prompt that expresses in language the region indicated by one or more points obtained by the sampling process described above.

[0050] (Loss calculation section 13) The loss calculation unit 13 calculates the loss by referring to the first gaze region and the second gaze region. As an example, the loss calculation unit 13 calculates the loss using a loss function configured such that the smaller the degree of difference between the first gaze region and the second gaze region, the smaller the value of the loss. The loss function referred to by the loss calculation unit 13 may also include the reconstruction loss, which shows the difference between the target image TI and the reconstructed image RE. More specific examples of the loss function referred to by the loss calculation unit 13 will be described later.

[0051] (Learning Section 14) The learning unit 14 refers to the loss calculated by the loss calculation unit 13 and trains the first generation unit 11. For example, the learning unit 14: • Encoder used by the object representation extraction unit 112 • Extraction model used by the object representation extraction unit 112 • Decodes constituting the reconstruction unit 121 The learning unit 14 updates at least one of these parameters by referring to the loss calculated by the loss calculation unit 13. For example, the learning unit 14 updates these parameters so that the loss calculated by the loss calculation unit 13 becomes smaller.

[0052] (Output information generation unit 15) The output information generation unit 15 generates output information including information generated (extracted) by the control unit 10, and outputs the generated output information to the outside via the communication unit 30 or the input / output unit 40. For example, the output information generation unit 15 may be configured to generate output information including object representations extracted by the object representation extraction unit 112, and to supply said output information to the monitoring device 60 via the communication unit 30. The output information may also include reconstructed images RE generated by the reconstruction unit 121, or information generated by other configurations of the control unit 10.

[0053] (Specific configuration example and processing example 1) Next, with reference to Figures 4 and 5, a specific configuration example of the information processing device 1A and processing example 1 will be described. Figure 4 is a diagram showing a specific configuration example of the information processing device 1A and processing example 1 during the learning phase. Figure 5 is a schematic diagram showing an example of the processing flow by the information processing device 1A during the learning phase.

[0054] As shown in Figure 4, first, the target image TI is acquired by the acquisition unit 111 of the first generation unit 11. The acquired target image TI is input to the encoder 1121 of the object representation extraction unit 112. Then, the object representation extraction model (extraction model) 1122 refers to the output of the encoder 1121 and extracts one or more object representations OR. The object representation extraction unit 112 also generates K (K is a natural number) first gaze region maps (attention masks in this example) based on the one or more object representations OR, and supplies the generated first gaze region maps to the prompt generation unit 122 and the gaze region loss calculation unit 141.

[0055] Furthermore, the one or more object representations OR extracted by the extraction model 1122 are supplied to the decoder (reconstruction unit) 121, and the decoder 121 generates (reconstructs) a reconstructed image RE from the object representations OR.

[0056] The target image TI is also supplied to the base model BM, and the second generation unit 12 uses the base model BM to generate K' (K' is a natural number) second gaze region maps from the target image TI and prompt PT, and supplies the generated second gaze region maps to the gaze region loss calculation unit 141.

[0057] (Step S11) As shown in Figure 5, in step S11, the target image TI is input to the encoder 1121 of the first generation unit 11. Then, the object representation extraction model 1122 of the first generation unit 11 refers to the output of the encoder 1121 and extracts one or more (K) object representations OR. Figure 5 schematically shows the first attention regions RI11 and RI12 indicated by the first attention region map (attention mask) generated by the first generation unit 11 in this step.

[0058] (Step S12) Step S12 in Figure 5 includes step S121, which generates one or more reconstructed regions RS reconstructed from the object representation OR; step S122, which binarizes each of the one or more reconstructed regions RS based on a defined threshold; step S123, which generates a prompt PT from the binarized reconstructed regions; and step S124, which generates high-precision second gaze regions RI21 and RI22 from the target image TI and prompt PT using the base model BM.

[0059] (Step S121) In step S121, the reconstruction unit (decoder) 121 generates one or more (K) reconstruction regions RS (decoder masks) reconstructed from the object representation OR. As an example, K (K is a natural number) reconstruction region maps (decoder masks in this example) are generated by referencing the K object representation ORs extracted by the object representation extraction unit 112 in step S11.

[0060] (Step S122) Step S122 binarizes each of the reconstructed one or more reconstructed regions RS based on a predetermined threshold. For example, K (where K is a natural number) reconstructed region maps generated in step S121 are each binarized based on a predetermined threshold.

[0061] (Step S123) Step S123 generates prompt PT from the binarized reconstructed regions. As an example, it generates rectangular prompts for K (where K is a natural number) reconstructed region maps that were binarized in step S122.

[0062] (Step S124) Step S124 generates high-precision second gaze regions RI21 and RI22 from the target image TI and prompt PT using the base model BM.

[0063] (Step S13) Then, in step S13, the gaze region loss calculation unit 141 shown in Figure 4 calculates the loss by referring to the first gaze region and the second gaze region. As an example, the loss calculation unit 13, as shown in step S13 of Figure 5, Differences between the first gaze region RI11 and the second gaze region RI21 Differences between the first gaze region RI12 and the second gaze region RI22 The loss (gaze region loss) is calculated using a loss function configured such that the smaller the value of the function, the smaller the loss value. Alternatively, the L2 norm may be used as the loss function. The gaze region loss calculated by the gaze region loss calculation unit 141 is supplied to the parameter update unit (learning unit) 14.

[0064] (Step S14) Then, in step S14, the learning unit 14 trains the extraction model by referring to the loss calculated in step S13. As an example, the learning unit 14 calculates the reconstruction loss which shows the difference between the target image TI and the reconstructed image RE, • The gaze area loss calculated in step S13, • The above reconstruction loss calculated in this step and So that the linear sum of becomes smaller, • Encoder used by the object representation extraction unit 112 • Extraction model used by the object representation extraction unit 112 • Decodes constituting the reconstruction unit 121 Update at least one of the parameters.

[0065] The learning unit 14 then refers to the updated parameters and determines whether the parameters have converged. If it determines that the parameters have converged, it terminates the learning process. If it determines that the parameters have not converged, it returns to step S11 and repeats the learning process.

[0066] Furthermore, a pre-trained model may be used as at least one of the encoder used by the object representation extraction unit 112, the extraction model used by the object representation extraction unit 112, and the decoder constituting the reconstruction unit 121. In such a case, the learning unit 14 will perform training on these pre-trained models using the process described above.

[0067] Furthermore, in the above example, the attention region indicated by the attention mask was used as the first attention region, but this is not limited to this. The learning process described above may also be performed using the attention region indicated by the decoder mask as the first attention region. Thus, in this example, • A process that uses the gaze region indicated by the attention mask as the first gaze region to train each model of the object representation extraction unit 112. • A process that uses the gaze region indicated by the attention mask as the first gaze region to train the decoder constituting the reconstruction unit 121. • A process in which the object representation extraction unit 112 learns each model using the gaze region indicated by the decoder as the first gaze region. • A process that uses the gaze region indicated by the decoder mask as the first gaze region to train the decoders that make up the reconstruction unit 121. You may perform any of the following actions.

[0068] (Specific configuration example and processing example 2) Next, with reference to Figure 6, a specific configuration example of the information processing device 1A and processing example 2 will be described. Figure 6 is a diagram showing a specific configuration example of the information processing device 1A and processing example 2 during the learning phase.

[0069] As shown in Figure 6, in this example, the decoder (reconstruction unit) 121 supplies the reconstructed image RE, reconstructed from the object representation OR, to the base model BM. On the other hand, in this example, the target image TI is not supplied to the base model BM. Other processing is the same as in the configuration example and processing example 1 described above, so redundant explanations are omitted. With the above configuration, the second gaze region can be divided with high accuracy using an extraction model learned using the knowledge from the pre-trained base model BM.

[0070] (Specific configuration example and processing example 3) Next, with reference to Figure 7, a specific configuration example of the information processing device 1A and processing example 3 will be described. Figure 7 is a diagram showing a specific configuration example of the information processing device 1A and processing example 3 during the learning phase.

[0071] As shown in Figure 7, in this example, the second generation unit 12 comprises a decoder (reconstruction unit) 121, a prompt generation unit 122, a pre-processing unit 125, a post-processing unit 126, and a base model. In this processing example, the configuration may include only one of the pre-processing unit 125 or the post-processing unit 126.

[0072] The preprocessing unit 125 applies preprocessing to the first attention region RI1 (attention mask) generated by the object representation extraction unit 112, or the reconstructed region RS (reconstructed region map, decoder mask) generated by the reconstruction unit 121, and supplies the preprocessed data to the prompt generation unit 122. For example, the preprocessing unit 125 applies preprocessing to the attention mask or decoder mask so that each mask (attention region) for each object representation (slot) OR is exclusive, and supplies the preprocessed attention mask or decoder mask to the prompt generation unit 122. For example, the preprocessing unit 125 may be configured to perform the above preprocessing by referring to the score (α coefficient) in the attention mask or decoder mask. In this way, by performing processing that depends on the state of other slots as preprocessing for the slot in question, attention regions related to each slot (object representation) can be generated more favorably.

[0073] The post-processing unit 126 applies post-processing to the second gaze region generated by the base model BM and supplies the post-processed data to the gaze region loss calculation unit 141. As an example, the post-processing unit 126 may perform a process to reshape the second gaze region generated by the base model BM into a desired shape.

[0074] Alternatively, the post-processing unit 126 may, for example, apply post-processing to the second attention region so that each mask (attention region) for each object representation (slot) OR becomes exclusive, and supply the second attention region after post-processing to the attention region loss calculation unit 141. For example, the pre-processing unit 125 may be configured to perform the pre-processing by referring to the score (α coefficient) in the second attention region. Alternatively, the post-processing unit 126 may be configured to perform the post-processing by referring to the score (α coefficient) in the attention mask or decoder mask. In this way, by performing processing that depends on the state of other slots as the post-processing, attention regions for each slot (object representation) can be generated more favorably.

[0075] Other processes are the same as those described in the configuration example and processing example 1, so redundant explanations will be omitted. With the configuration of this example, it is possible to generate a highly accurate model by accurately dividing the second gaze region and training a model for extracting object representations.

[0076] (Specific Configuration Example and Processing Example 4) Next, with reference to Figure 8, a specific configuration example and processing example 4 of the information processing device 1A will be described. This example is a specific configuration example and processing example of the information processing device 1A in the inference phase. Figure 8 is a diagram showing a specific configuration example and processing example of the information processing device 1A related to this example.

[0077] As shown in Figure 8, in the inference phase, the target image TI to be inferred is first acquired by the acquisition unit 111. This target image TI is, for example, captured by the camera group 70 shown in Figure 3. However, this is not limited to this example.

[0078] The target image TI acquired by the acquisition unit 111 is supplied to the encoder 1121. Then, the object representation extraction model (extraction model) 1122, which has been learned by any of the processing examples 1 to 3 described above, refers to the output of the encoder 1121 and extracts one or more object representations OR. The extracted object representations are supplied to the output information generation unit 15 as an example and included in the output information. The output information including the object representations is supplied to the monitoring device 60 as an example and referred to in the monitoring process. Note that the information output in the inference phase is not limited to the above example, and the output image may also include the reconstructed image RE generated by the decoder 121 by referring to the object representations.

[0079] With the above configuration, object representations can be suitably extracted from the target image TI using an extraction model that has been efficiently trained using knowledge from a pre-trained base model BM.

[0080] (Additional information for Exemplary Embodiment 2) This exemplary embodiment is not limited to the examples described above. For example, a pre-trained model may be used as at least one of the encoder 1121 and the object representation extraction model (extraction model) 1122. For example, a model pre-trained with a large amount of training data may be used as at least one of the encoder 1121 and the object representation extraction model 1122, and then the learning process may be advanced by transferring the knowledge of the base model BM as described in the processing example above. By starting the learning process with a pre-trained model that can roughly estimate the object region in this way, the second gaze region can be divided with high accuracy, and the model for extracting object representations can be trained more efficiently.

[0081] [Third Embodiment] A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiments are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0082] (Configuration of Information Processing System 100B) The configuration of the information processing system 100B according to this exemplary embodiment will be described with reference to Figure 9. Figure 9 is a block diagram showing the configuration of the information processing system 100B. As shown in Figure 9, the information processing system 100B comprises an information processing device 1A and a robot 80 connected to the information processing device 1A via a network N. The information processing device 1A and the network N are the same as in the exemplary embodiment 1, so redundant explanations will be omitted.

[0083] (Robot 80) As shown in Figure 9, the robot 80 comprises a control unit 81, a communication unit 82, a camera group 83, and a drive unit 84. The camera group 83 comprises one or more cameras that capture images of the area in which the robot 80 operates. The drive unit 84 comprises a robot arm, wheels, etc., which are driven based on control by the control unit 81.

[0084] The communication unit 82 supplies images captured by the camera group 83 to the information processing device 1A via the network N. The communication unit 82 also receives object representations extracted from the images from the information processing device 1A via the network N and supplies them to the control unit 81.

[0085] The control unit 81 supplies images of one or more objects to the information processing device 1A via the communication unit 82, and controls the drive unit 84 by referring to the object representation extracted by the information processing device 1A from the images. For example, the control unit 81 refers to the object representation acquired via the communication unit 82 to perform hidden area estimation of the world model, future prediction processing, etc., and learns and plans the control of the robot arm.

[0086] (Information Processing Device 1A) As partially described above, in this exemplary embodiment, images captured by the camera group 83 are supplied to the information processing device 1A and are referenced as target images TI in at least one of the learning phase and inference phase described in exemplary embodiment 2.

[0087] Then, in the inference phase, the information processing device 1A extracts object representations from the target image TI by performing inference processing using the image captured by the camera group 70 as the target image TI. The extracted object representations are then supplied to the robot 80 and referenced to control the drive unit 84.

[0088] [Examples of implementation using software] Some or all of the functions of the information processing devices 1,1A (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0089] In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 10. Figure 10 is a block diagram showing the hardware configuration of computer C, which functions as each of the above devices.

[0090] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P that causes computer C to operate as each of the above-mentioned devices. In computer C, processor C1 reads program P from memory C2 and executes it, thereby realizing each of the above-mentioned devices.

[0091] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0092] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0093] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0094] Furthermore, each of the above functions of each of the above devices may be implemented by a single processor in a single computer, by multiple processors in a single computer working together, or by multiple processors in each of multiple computers working together. In addition, the programs for implementing each of the above functions in each of the above devices may be stored in a single memory in a single computer, distributed and stored in multiple memories in a single computer, or distributed and stored in multiple memories in each of multiple computers.

[0095] (Note A1) A first generation means that extracts object representations of each object contained in the target image and generates one or more first gaze regions based on said object representations, A second generation means that generates a prompt based on the first gaze region and generates one or more second gaze regions from the target image and the prompt using a pre-trained model, Loss calculation means for calculating the loss by referring to the first gaze region and the second gaze region, A learning means that causes the first generation means to learn by referring to the loss An information processing device equipped with the following features.

[0096] (Appendix A2) The first generation means is, The system includes an object representation extraction means for extracting object representations of each object by referring to the aforementioned target image, The second generation means is The system includes reconstruction means for reconstructing each of the aforementioned objects by referring to the object representation of each of the aforementioned objects, The learning means updates at least the parameters of the object representation extraction means and the reconstruction means by referring to the loss. The information processing device described in Appendix A1.

[0097] (Note A3) The object representation extraction means generates the first gaze region The information processing device described in Appendix A2.

[0098] (Note A4) The reconstruction means generates one or more reconstruction regions reconstructed from the object representation. The information processing device described in Appendix A2.

[0099] (Note A5) The object representation extraction means is An encoder into which the aforementioned target image is input, An extraction model that extracts object representations of each object by referring to the output of the encoder, It is equipped with The information processing device described in Appendix A2.

[0100] (Note A6) The loss calculated by the loss calculation means is The value of the loss is set to decrease as the degree of difference between the first gaze region and the second gaze region decreases. An information processing device as described in any one of the appendices A1 to A5.

[0101] (Note A7) The second generation means is A binarization means that generates the one or more reconstructed regions by referring to each of the one or more reconstructed regions, Prompt generation means that generates one or more prompts based on each of the one or more binarized reconstruction regions, A gaze region estimation means for estimating one or more second gaze regions from the target image and the pre-trained model into which the prompt has been input. It is equipped with An information processing device as described in any one of the appendices A1 to A5.

[0102] (Note A8) The second generation means is The reconstructed image reconstructed by the reconstruction means and the pre-trained model to which the prompt has been input are used to estimate the one or more second gaze regions. An information processing device as described in any one of the appendices A1 to A5.

[0103] (Note A9) One or more processors, The object representation of each object contained in the target image is extracted using an extraction model, and one or more first gaze regions are generated based on these object representations. A prompt is generated based on the first gaze region, and one or more second gaze regions are generated from the target image and the prompt using a pre-trained model. The loss is calculated by referring to the first gaze region and the second gaze region, The extraction model is trained by referring to the loss. An information processing method that includes this.

[0104] (Note A10) A program that makes a computer function as an information processing device. The aforementioned computer, A first generation means that extracts object representations of each object contained in the target image and generates one or more first gaze regions based on said object representations, A second generation means that generates a prompt based on the first gaze region and generates one or more second gaze regions from the target image and the prompt using a pre-trained model, Loss calculation means for calculating the loss by referring to the first gaze region and the second gaze region. A learning means for causing the first generation means to learn by referring to the loss, A program that makes it function as such. [Explanation of symbols]

[0105] 1,1A Information Processing Device 100A, 100B Information Processing System 11. First generation unit (first generation means) 12. Second generation unit (second generation means) 13 Loss calculation section (loss calculation means) 14. Learning Section (Learning Methods)

Claims

1. A first generation means that extracts object representations of each object included in the target image and generates one or more first gaze regions based on said object representations, A second generation means that generates a prompt based on the first gaze region and generates one or more second gaze regions from the target image and the prompt using a pre-trained model, Loss calculation means for calculating the loss by referring to the first gaze region and the second gaze region, A learning means for causing the first generation means to learn by referring to the loss An information processing device equipped with the following features.

2. The first generating means is The system includes an object representation extraction means for extracting object representations of each object by referring to the aforementioned target image, The second generation means is, The system includes reconstruction means for reconstructing each of the aforementioned objects by referring to the object representation of each of the aforementioned objects, The learning means updates at least the parameters of the object representation extraction means and the reconstruction means by referring to the loss. The information processing apparatus according to claim 1.

3. The object representation extraction means generates the first gaze region. The information processing apparatus according to claim 2.

4. The reconstruction means generates one or more reconstruction regions reconstructed from the object representation. The information processing apparatus according to claim 2.

5. The object representation extraction means is An encoder into which the aforementioned target image is input, An extraction model that extracts object representations of each object by referring to the output of the encoder, It is equipped with The information processing apparatus according to claim 2.

6. The loss calculated by the loss calculation means is The value of the loss is set to decrease as the degree of difference between the first gaze region and the second gaze region decreases. The information processing apparatus according to any one of claims 1 to 5.

7. The second generation means is, A binarization means that generates the one or more reconstructed regions by referring to each of the one or more reconstructed regions, Prompt generation means for generating one or more prompts based on each of the one or more binarized reconstruction regions, A gaze region estimation means for estimating one or more second gaze regions from the target image and the pre-trained model into which the prompt has been input. It is equipped with The information processing apparatus according to any one of claims 1 to 5.

8. The second generation means is, The reconstructed image reconstructed by the reconstruction means and the pre-trained model to which the prompt has been input are used to estimate the one or more second gaze regions. The information processing apparatus according to any one of claims 1 to 5.

9. One or more processors, The object representation of each object contained in the target image is extracted using an extraction model, and one or more first gaze regions are generated based on said object representation. A prompt is generated based on the first gaze region, and one or more second gaze regions are generated from the target image and the prompt using a pre-trained model. The loss is calculated by referring to the first gaze region and the second gaze region, The extraction model is trained by referring to the loss. An information processing method that includes this.

10. A program that makes a computer function as an information processing device. The aforementioned computer, A first generation means that extracts object representations of each object included in the target image and generates one or more first gaze regions based on said object representations, A second generation means that generates a prompt based on the first gaze region and generates one or more second gaze regions from the target image and the prompt using a pre-trained model, Loss calculation means for calculating the loss by referring to the first gaze region and the second gaze region. A learning means for causing the first generation means to learn by referring to the loss, A program that makes it function as such.

Images