Information processing device, information processing program, and information processing method

Abstract

An image feature vector is extracted from image data, and an image mask is generated from the image feature vector according to feature weights. A masked image feature vector is generated by applying the image mask to the image feature vector. Text information related to the content of the image data includes supplementary text information representing clothing of a person; a text mask is applied to the supplementary text information to generate masked text information. A masked text feature vector is extracted from the masked text information. A machine learning model (30) is trained on the basis of the masked image feature vector and the masked text feature vector, thereby enabling person re-identification that is not affected by clothing.

Description

Information processing device, information processing program, and information processing method

[0001] The present invention relates to an information processing device, an information processing program, and an information processing method.

[0002] Person re-identification is known as a method for determining whether a person shown in one image is the same person as a person shown in another image.

[0003] It is generally known that high accuracy can be achieved in person re-identification by using a person's appearance characteristics. However, because appearance characteristics depend on clothing, accuracy decreases in environments where clothing changes between people, or on different days, or in environments where everyone wears the same uniform, such as factories or schools.

[0004] Therefore, a method for extracting clothing-independent features is required. Examples of clothing-independent features include facial features, body shape, and gait.

[0005] Recently, a person re-identification method using a VLM (Vision Language Model) has become known (see Non-Patent Document 1 below). In this person re-identification method using a VLM, recognition accuracy can be improved by inputting text information (prompts) indicating person characteristics (gender, age, clothing) along with an image.

[0006] JP 2022-177220 A JP 2023-7419 A

[0007] Siyuan Li, Li Sun, Qingli Li, "CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labels," [online], November 25, 2022, AAAI Conference on Artificial Intelligence, [Retrieved August 6, 2024], Internet <URL: https: / / arxiv.org / abs / 2211.13977>

[0008] However, in the person re-identification method using the above-mentioned VLM, clothing information is included in the text, and therefore clothing-dependent features (clothing-dependent features) are extracted.

[0009] In one aspect, the present invention aims to enable person re-identification that is not affected by clothing.

[0010] For this reason, this information processing device extracts image features from image data, generates an image mask from the image features according to the weights of the features, generates masked image features by applying the image mask to the image features, generates masked text information by applying a text mask to additional text information that represents the clothing of a person included in text information related to the content of the image data, extracts masked text features from the masked text information, and trains a machine learning model based on the masked image features and the masked text features.

[0011] According to one embodiment, person re-identification can be achieved that is not affected by clothing.

[0012] FIG. 1 is a diagram schematically illustrating a configuration of an association device according to an embodiment; FIG. 2 is a block diagram illustrating an example of a hardware (HW) configuration of a computer that realizes the functions of an association device according to an embodiment; FIG. 3 is a diagram illustrating an overview of processing in a training phase of an association device according to an embodiment; FIG. 4 is a diagram illustrating an overview of processing in a training phase of an association device according to an embodiment; FIG. 5 is a flowchart for explaining processing in a training phase of an association device according to an embodiment; FIG. 6 is a flowchart for explaining processing in an inference phase of an association device according to an embodiment; and FIG. 7 is a diagram for explaining a modified example of processing by an image mask generation unit of an association device according to an embodiment.

[0013] Hereinafter, embodiments of the present information processing device, information processing program, and information processing method will be described with reference to the drawings. However, the embodiments shown below are merely examples, and are not intended to exclude the application of various modifications and techniques not explicitly stated in the embodiments. In other words, the present embodiment can be implemented with various modifications (such as combining the embodiment and each modification) within the scope of its intent. Furthermore, each figure does not intend to include only the components shown in the figure, but may include other functions, etc.

[0014] (A) Configuration Fig. 1 is a diagram schematically illustrating the configuration of a matching device 1 according to an embodiment. The matching device 1 is a device that realizes person re-identification, which determines whether a person appearing in a first image captured by a first camera and a person appearing in a second image captured by a second camera are the same person. The first camera and the second camera may be the same or different.

[0015] In this embodiment, the matching device 1 determines whether a person appearing in an image (corresponding to a second image) taken by a camera (not shown in the figure) (corresponding to a second camera) is the same as a person registered in a person registration database (DB) 25.

[0016] 2 is a block diagram showing an example of the hardware (HW) configuration of a computer 10 that realizes the functions of the association device 1 according to an embodiment. When multiple computers are used as HW resources that realize the functions of the association device 1, each computer may have the HW configuration shown in FIG.

[0017] (A-1) Hardware Configuration As shown in FIG. 2, the computer 10 may, as its HW configuration, illustratively include a processor 10a, a GPU 10b, a memory 10c, a storage unit 10d, an IF (Interface) unit 10e, an IO (Input / Output) unit 10f, and a reading unit 10g.

[0018] The processor 10a is an example of a control unit that performs various controls and calculations. The processor 10a may be connected to each block in the computer 10 via a bus 10j so that they can communicate with each other. The processor 10a may be a multiprocessor including multiple processors, a multi-core processor having multiple processor cores, or a configuration having multiple multi-core processors.

[0019] Examples of the processor 10a include integrated circuits (ICs) such as a CPU, MPU, APU, DSP, ASIC, and FPGA. Note that the processor 10a may be a combination of two or more of these integrated circuits. CPU is an abbreviation for Central Processing Unit, MPU is an abbreviation for Micro Processing Unit, APU is an abbreviation for Accelerated Processing Unit, DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0020] The GPU 10b may be, for example, an accelerator such as a general-purpose computing on graphics processing unit (GPGPU). The GPU 10b may also be used to control screen display on an output device such as a monitor in the IO unit 10f. The GPU 10b may be configured as an accelerator that executes machine learning processing and inference (prediction) processing using a machine learning model. It can be said that the GPU 10b has higher processing performance in machine learning (training) processing and inference processing than the processor 10a.

[0021] The memory 10c is an example of HW that stores various types of data, programs, and other information. The memory 10c may be, for example, a volatile memory such as a dynamic random access memory (DRAM) or a non-volatile memory such as a persistent memory (PM), or both.

[0022] The storage unit 10d is an example of HW that stores various types of data, programs, and other information. Examples of the storage unit 10d include various storage devices such as a magnetic disk device such as a hard disk drive (HDD), a semiconductor drive device such as a solid state drive (SSD), and a nonvolatile memory. Examples of nonvolatile memory include a flash memory, a storage class memory (SCM), and a read-only memory (ROM).

[0023] The storage unit 10d may store a program 10h (information processing program) that realizes all or part of the various functions of the computer 10.

[0024] For example, the processor 10a of the association device 1 can implement the functions of the inference unit 2 and the training unit 3 shown in FIG. 1 by expanding the program 10h stored in the storage unit 10d into the memory 10c and executing it.

[0025] The IF unit 10e is an example of a communication IF that controls connections and communications between the computer 10 and other computers, etc. For example, the IF unit 10e may include an adapter that complies with a LAN (Local Area Network) such as Ethernet (registered trademark) or optical communications such as FC (Fibre Channel). The adapter may support either or both wireless and wired communication methods.

[0026] For example, the association device 1 may be connected to an image database (not shown) via the IF unit 10e and a network so as to be able to communicate with each other. The program 10h may be downloaded from the network to the computer 10 via the communication IF and stored in the storage unit 10d.

[0027] The IO unit 10f may include one or both of an input device and an output device. Examples of input devices include a keyboard, a mouse, and a touch panel. Examples of output devices include a monitor, a projector, and a printer. The IO unit 10f may also include a touch panel that combines an input device and a display device. The output device may be connected to the GPU 10b.

[0028] The reading unit 10g is an example of a reader that reads data and program information recorded on the recording medium 10i. The reading unit 10g may include a connection terminal or device to which the recording medium 10i can be connected or inserted. Examples of the reading unit 10g include an adapter compliant with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. Note that the recording medium 10i may store the program 10h, and the reading unit 10g may read the program 10h from the recording medium 10i and store it in the memory unit 10d.

[0029] Examples of the recording medium 10i include non-transitory computer-readable recording media such as magnetic / optical disks and flash memories. Examples of magnetic / optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of flash memories include semiconductor memories such as USB memories and SD cards.

[0030] The above-described HW configuration of the computer 10 is an example. Therefore, the HW in the computer 10 may be increased or decreased (for example, adding or deleting any block), divided, integrated in any combination, or the HW may be added or deleted as needed.

[0031] (A-2) Functional Configuration As shown in FIG. 1, the association device 1 may illustratively have functions as an inference unit 2 and a training unit 3. The inference unit 2 functions in the inference phase, and the training unit 3 functions in the training phase. These functions may be realized by the hardware of a computer 10 (see FIG. 2).

[0032] 1, the training unit 3 has functions as an image input unit 31, an image feature extraction unit 32, an image mask generation unit 33, an image mask processing unit 34, a text input unit 35, a text feature extraction unit 36, a text mask processing unit 37, a loss calculation unit 38, and a training data DB 39. Hereinafter, the image feature extraction unit 32 may be referred to as an image encoder 32. Also, the text feature extraction unit 36 ​​may be referred to as a text encoder 36.

[0033] A training data set is input to the training unit 3. The training data set includes image data (training image data), a prompt corresponding to the image data, and a person ID (person_id) and clothing ID (clothes_id) corresponding to the image data. The clothes_id can be called a clothes label.

[0034] The image data contains a person. The person ID is identification information that identifies the person in the image data, and the clothing ID is identification information that identifies the clothing of this person. The prompt is text data that represents information about the person in the image data. An example of a prompt is shown below.

[0035] A photo of [X]_1 [X]_2,… [X]_M person This prompt shows an example where the image data is a photograph. "[X]_1 [X]_2,… [X]_M" included in this prompt can be called person information tokens. In person information tokens, M is the number of tokens.

[0036] 3 and 4 are diagrams showing an overview of the processing in the training phase of the association device 1 according to an embodiment. The training phase has Stage 1 and Stage 2. Fig. 3 shows the processing in Stage 1, and Fig. 4 shows the processing in Stage 2.

[0037] In stage 1, clothing information tokens are added to the prompt to generate an extended prompt. The clothing information tokens are an example of additional text information that describes the clothing of a person depicted in the image data. This process is particularly implemented by the text input unit 35. An example of an extended prompt is shown below.

[0038] A photo of [X]_1 [X]_2,... [X]_M person wearing [Y]_1 [Y]_2,... [Y]_M. The "[Y]_1 [Y]_2,... [Y]_M" included in this extended prompt is an example of a clothing information token. In this embodiment, for example, M=4 may be set as the best parameter.

[0039] Also, as shown in Figure 3, in stage 1, supervised learning is performed on the person information tokens and clothing information tokens in the extended prompt based on the person ID (person_id) and clothing ID (clothes_id) provided from the training dataset.

[0040] The image encoder 32 (image feature extraction unit 32) extracts features from the image data (f ＩIn addition, the text encoder 36 (text feature extraction unit 36) extracts features in response to the extended prompt (f Ｔ ).

[0041] Then, the image-to-text (i2t) contrast loss (i2t Loss) L i2t and the text-to-image (t2i) contrast loss (t2i Loss) L t2i 3, the parameters of the text encoder 36 are updated and the text (tokens) are optimized (leanable) as indicated by the unlocked key mark attached to the text encoder 36. Also, the fact that the parameters of the image encoder 32 are not updated (frozen) is indicated by the locked key mark.

[0042] The function of stage 1 may be realized by the machine learning model 30. The machine learning model 30 may be a VLM model.

[0043] In stage 2, a masking process (prompt mask) is performed on the extended prompt, masking (setting to 0) the clothing information token set in stage 1. This masking process can eliminate clothing information contained in the extended prompt (text). This process is realized, for example, by the text masking processor 37.

[0044] Furthermore, in stage 2, a Cloth Classifier is added. The Cloth Classifier is a classifier that identifies clothing in an image. A known classifier may be used as the Cloth Classifier, and a description thereof will be omitted.

[0045] In stage 2, the gradient between the image feature extracted by the image encoder 32 and the image feature extracted from the image data by the Cloth Classifier is calculated, and a masking process is performed to mask (set to 0) areas of the image feature extracted by the image encoder 32 where this gradient is large. These areas of large gradient represent areas of interest to the Cloth Classifier and can be considered to represent clothing information. This masking process can eliminate clothing information contained in the image feature. Note that the areas with large gradients may be, for example, the top p% (e.g., p = 33.3). These processes are implemented, for example, by the image mask generation unit 33 and the image mask processing unit 34.

[0046] 4, the fact that the parameters of the image encoder 32 are being updated is indicated by an unlocked key mark attached to the image encoder 32. Also, the fact that the parameters of the text encoder 36 are not being updated (frozen) is indicated by a locked key mark.

[0047] Also, in stage 2, the text cross entropy L i2tce and triplet loss L tri and ID loss L id The text cross entropy L i2tce and triplet loss L tri and ID loss L id More on this later.

[0048] Furthermore, in stage 2, the loss due to the error between the Cloth Classifier and the clothing ID (clothes_id) (Cloth Classification Loss) L Ｃ and a clothing-based adversarial loss (Cloth-based adversarial loss) L ＣＡ The function of stage 2 is realized by the machine learning model 30 described above.

[0049] The training data DB 39 stores a training data set used for training (machine learning) the model included in the association device 1. The training data set includes, for example, image data in which a person appears, and a person label and a clothing label (index) related to the person appearing in the image data.

[0050] The image input unit 31 captures image data from the training data set. The captured training image data is stored in a predetermined storage area of ​​the memory 10c or the storage unit 10d.

[0051] The image feature extraction unit 32 extracts features (image features) from image data. The function of the image feature extraction unit 32 can be realized by a known method, and a description thereof will be omitted. The function of the image feature extraction unit 32 may be realized by a machine learning model 30.

[0052] The image mask generation unit 33 generates a weight mask according to the gradient strength of the image feature extracted by the image feature extraction unit 32. The weight mask generated by the image mask generation unit 33 may be referred to as an image mask. The image mask generation unit 33 generates an image mask from the image feature according to the weight of the feature.

[0053] The image mask generation unit 33 calculates the gradient between the image features extracted by the image feature extraction unit 32 and the image features extracted from the image data by the Cloth Classifier, and performs masking processing to mask (set to 0) areas of the image features extracted by the image feature extraction unit 32 where this gradient is large.

[0054] The mask generation process by the image mask generation unit 33 is exemplified by the following equations (1) and (2): Equation (1) represents the gradient gf between the image feature extracted by the image feature extraction unit 32 and the image feature extracted from the image data by the Cloth Classifier.

[0055]

[0056] The area with a large gradient can be said to represent the area that the Cloth Classifier is focusing on, and can be considered to represent clothing information. Note that the area with a large gradient can be determined by, for example, sorting the areas in descending order of gradient and selecting the top p% (e.g., p=33.3).

[0057] In the formula (2), m(i) represents a masking vector. In the example shown in the formula (2), the gradient gf(i) is ｐ This indicates that a mask is generated that sets the above regions to 0. The image mask processing unit 34 applies the mask generated by the image mask generation unit 33 to the image feature amount. The following equation (3) represents the processing of the image mask processing unit 34.

[0058]

[0059] The image mask processing unit 34 performs an element-wise product of the image features with the generated mask.

[0060] The image mask processing unit 34 generates masked image features by applying an image mask to the image features.

[0061] The text input unit 35 acquires text (prompt) from the training dataset. The text input unit 35 generates an extended prompt by adding the above-described clothing information token to the prompt. The text input unit 35 generates the clothing information token based on, for example, a clothes label (index) included in the training dataset. The clothes label (index) is an example of label information related to image data.

[0062] The text input unit 35 may, for example, add a clothing information token after a person information token in the prompt.

[0063] The text input unit 35 generates clothing information tokens (additional text information) based on label information associated with the image data and adds them to the prompt (text information).

[0064] The text mask processing unit 37 masks (sets to 0) the clothing information tokens added by the text input unit 35 in the extended prompt, thereby removing the clothing information tokens from the prompt. An extended prompt with a mask (text mask) applied to the clothing information tokens is an example of masked text information. An example of masking the clothing information tokens in an extended prompt is shown below.

[0065] A photo of [X]_1 [X]_2,… [X]_M person wearing [0]_1 [0]_2,… [0]_M The text mask processing unit 37 generates masked text information by applying a text mask to clothing information tokens (additional text information representing clothing) included in the prompt (text information related to the content of the image data).

[0066] The text feature extraction unit 36 ​​extracts features (masked text features) from the extended prompt (masked text information) in which a mask (text mask) has been applied to the clothing information token by the text mask processing unit 37. The function of the text feature extraction unit 36 ​​may be realized by the machine learning model 30.

[0067] The loss calculation unit 38 calculates a loss function including the loss of the masked image feature, the loss of the masked text feature, and the loss of person re-identification.

[0068] In stage 1, the loss calculator 38 calculates the image-to-text (i2t) contrast loss L i2t and the contrast loss L from text to image (t2i: text-to-image) t2i Calculate the relative loss L i2t An example of the calculation formula is shown in the following formula (4), and the relative loss L t2i An example of the calculation formula is shown in the following formula (5).

[0069]

[0070] The loss calculation unit 38 calculates the relative loss L i2t and the contrast loss L t2i Adding these together, the loss in stage 1, L stage1 Calculate the loss Lstage1 An example of the calculation formula is shown in the following formula (6).

[0071] L stage1 =L i2t +L t2i ...(6) In addition, the loss calculation unit 38 calculates the ID loss L id and triplet loss L tri and the text cross entropy L i2tce and loss L C and hostile loss L CA and asks for.

[0072] ID loss L id may be calculated using, for example, the following formula (7): tri The text cross entropy L may be calculated using, for example, the following equation (8): i2tce The loss L may be calculated using, for example, the following equation (9): C The adversarial loss L may be calculated using, for example, the following equation (10). CA may be calculated using, for example, the following formula (11):

[0073] Then, the loss calculation unit 38 calculates the loss L using the following equation (12): stage2,step1 Calculate.

[0074] L stage2,step1 =L id +L tri +L i2tce +L C ...(12) Loss L calculated by the loss calculation unit 38 stage2,step1 The process of calculating the loss L calculated in step 1 can be called step 1. stage2,step1 is used for training (machine learning) by the training unit 3 to make the Cloth Classifier identify clothing.

[0075] Furthermore, the loss calculation unit 38 calculates the loss L using the following equation (13): stage2,step2 Calculate.

[0076] L stage2,step2 =L id +L tri +L i2tce +LCA ... (13) Loss L calculated by the loss calculation unit 38 stage2,step2 The process of calculating the loss L calculated in step 2 can be called step 2. stage2,step2 is used for training (machine learning) by the training unit 3 to prevent the Cloth Classifier from identifying clothing.

[0077] In addition, triplet loss L tri , text cross entropy L i2tce , ID loss L id , loss L C , adversarial loss L CA is an example of a loss due to estimation based on the masked image features. Also, the text cross entropy L i2tce is an example of a loss due to estimation based on masked text features.

[0078] The training unit 3 trains the machine learning model 30 using the losses calculated by the loss calculation unit 38.

[0079] In stage 1, the training unit 3 calculates the loss L stage1 The person information tokens and clothing information tokens are optimized using the following algorithms. The optimization of these tokens may be performed using various known algorithms such as the steepest descent method.

[0080] In addition, in stage 2, the training unit 3 optimizes the Image Encoder, ID Classifier, and Cloth Classifier in two steps (steps 1 and 2).

[0081] In step 1, the training unit 3 calculates the loss L stage2,step1 The machine learning model 30 (Image Encoder) is trained (machine learning) using the above so that the Cloth Classifier can identify clothing.

[0082] In step 2, the training unit 3 calculates the loss L stage2,step2By using this method, the machine learning model 30 (image encoder) is trained so that the Cloth Classifier, which has improved classification accuracy, does not use adversarial learning to identify clothing, thereby enabling the machine learning model 30 (image feature extraction unit 23, image encoder) to focus on clothing-independent areas in the inference phase.

[0083] As shown in FIG. 1, the inference unit 2 has the functions of a camera photographing unit 21, a frame image set storage unit 22, an image feature extraction unit 23, a person association unit 24, a person registration DB 25, and a result output unit 26.

[0084] Information about people who appear in image data (corresponding to a first image) captured by a camera (corresponding to a first camera) is registered in person registration DB 25. For example, information about feature amounts generated based on the image data and information that identifies people who appear in the image data may be stored in association with each other in person registration DB 25, and various modifications can be made to the data.

[0085] The camera photographing unit 21 photographs the person to be inferred using a camera (not shown) to obtain image data (frame images) showing the person to be inferred. A plurality of frame images may be referred to as a frame image set. Note that instead of the camera photographing unit 21 photographing the person to be inferred, image data photographed by another camera may be input to the inference unit 2.

[0086] The frame image set storage unit 22 stores a set of frame images acquired by the camera photographing unit 21. The function of the frame image set storage unit 22 may be realized by, for example, the memory 10c or the storage unit 10d.

[0087] The image feature extraction unit 23 reads a frame image from the frame image set stored in the frame image set storage unit 22, and extracts image features of this frame image. The function of this image feature extraction unit 23 may be realized by the machine learning model 30. Particularly in the inference phase, the machine learning model 30 may be called an image feature extraction model.

[0088] The person association unit 24 uses the image features extracted by the image feature extraction unit 23 to select a person with the most similar features from the person registration DB 25 and associates the person with the frame image. For example, the person association unit 24 may compare each of the multiple image features registered in the person registration DB 25 with the image features extracted by the image feature extraction unit 23. Then, from the multiple image features registered in the person registration DB 25, the person most similar to the image feature extracted by the image feature extraction unit 23 may be selected.

[0089] The similarity of image features can be determined using, for example, vector similarity, or a distance system such as Euclidean distance or a direction system such as cosine similarity, and various modifications can be made.

[0090] The result output unit 26 outputs the person ID of the information of the person that the person associating unit 24 selected from the person registration DB 25 and associated with the frame image. The result output unit 26 may output the person ID to an output device such as a monitor (not shown), for example.

[0091] (B) Operation The processing in the training phase in the association device 1 according to the embodiment configured as described above will be described with reference to the flowchart (steps A1 to A4) shown in FIG.

[0092] In step A1, the training unit 3 reads parameters of the machine learning model 30. In step A2, the training unit 3 reads a training dataset. Specifically, the image input unit 31 retrieves image data from the training dataset, and the text input unit 35 retrieves a prompt from the training dataset. The text input unit 35 also generates an extended prompt by adding the above-mentioned clothing information token to the prompt.

[0093] In step A3, the image feature extraction unit 32 extracts image features from the image data, and the text feature extraction unit 36 ​​extracts features from the extended prompt (text).

[0094] In step A4, the loss calculation unit 38 calculates the loss function. i2t and contrast loss Lt2i Calculate the loss L stage1 is calculated (stage 1).

[0095] The training unit 3 calculates the loss L calculated by the loss calculation unit 38. stage1 Using this, the person information tokens and clothing information tokens are optimized (Stage 1).

[0096] In addition, the loss calculation unit 38 calculates the ID loss L id and triplet loss L tri and the text cross entropy L i2tce and loss L C and hostile loss L CA The loss calculation unit 38 calculates the ID loss L id and triplet loss L tri and the text cross entropy L i2tce and loss L C Using this, the loss L stage2,step1 (Step 1 of Stage 2).

[0097] The training unit 3 calculates the L stage2,step1 The machine learning model 30 is trained (machine learning) using the above data so that the Cloth Classifier can identify clothing (Step 1 of Stage 2).

[0098] Furthermore, the loss calculation unit 38 calculates the ID loss L id and triplet loss L tri and the text cross entropy L i2tce and hostile loss L CA Using this, the loss L stage2,step2 (Step 2 of Stage 2).

[0099] The training unit 3 calculates the L stage2,step2 The machine learning model 30 (image feature extraction unit 23) is trained so that the Cloth Classifier with improved classification accuracy does not use adversarial learning to classify clothing. Then, the process ends.

[0100] Next, the processing in the inference phase in the association device 1 according to the embodiment will be described with reference to the flowchart (steps B1 to B4) shown in FIG.

[0101] In step B1, the inference unit 2 reads parameters related to image feature extraction in the machine learning model 30 (image feature extraction model).

[0102] The camera photographing unit 21 photographs the person to be inferred using a camera, thereby acquiring image data (frame images). The photographed frame images are stored in the frame image set storage unit 22.

[0103] In step B 2 , the image feature extraction unit 23 reads a frame image from the frame image set stored in the frame image set storage unit 22 .

[0104] In step B3, the image feature extraction unit 23 extracts image feature values ​​from the read frame images.

[0105] In step B4, the person association unit 24 uses the image features extracted by the image feature extraction unit 23 to select a person with the closest features from the person registration DB 25 and associates the person with the frame image. The result output unit 26 outputs the person ID of the information on the person that the person association unit 24 selected from the person registration DB 25 and associated with the frame image. Thereafter, the processing ends.

[0106] (C) Effects As described above, according to the matching device 1 according to an example embodiment, in the training phase, the image mask generation unit 33 generates a weight mask according to the gradient strength of the image feature extracted from the image data by the image feature extraction unit 32. Then, the image mask processing unit 34 applies the mask generated by the image mask generation unit 33 to the image feature to generate masked image feature. As a result, the masked image feature does not include information about the clothing of the person appearing in the image data. The masked image feature is a clothing-independent feature.

[0107] Furthermore, the text mask processing unit 37 applies a text mask to the clothing information tokens added to the extended prompt by the text input unit 35 to generate masked text information. The text feature extraction unit 36 ​​extracts masked text features from the masked text information. As a result, the masked text features do not include information about the clothing of people appearing in the image data. The masked text features are clothing-independent features.

[0108] Then, the loss calculation unit 38 calculates a loss function including the loss of the masked image features, the loss of the masked text features, and the loss of person re-identification, and the training unit 3 trains the machine learning model 30 using the losses calculated by the loss calculation unit 38.

[0109] As a result, the machine learning model 30 is trained using clothing-independent features (masked image features and masked text features), thereby realizing person re-identification that is not affected by clothing in the inference phase.

[0110] In stage 1, the text input unit 35 adds a clothing information token representing the clothing of the person appearing in the image data to the prompt, and the text mask processing unit 37 masks this clothing information token, thereby removing the clothing information from the text features and easily generating clothing-independent features.

[0111] Furthermore, the image mask generation unit 33 calculates the gradient between the image features extracted by the image feature extraction unit 32 and the image features extracted from the image data by the Cloth Classifier, and the image mask processing unit 34 performs masking processing to set areas of the image features extracted by the image feature extraction unit 32 where this gradient is large to 0. This makes it possible to remove clothing information from the image features and easily generate clothing-independent features.

[0112] (D) Other The disclosed technology is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present embodiment. The configurations and processes of the present embodiment can be selected or combined as needed.

[0113] For example, in the above-described embodiment, an example is shown in which the image data is a photograph, and an extended prompt beginning with "A photo of" is used, but this is not limited to this. If the image data is a video, an extended prompt beginning with "A movie of" or "A video of" may be used instead of "A photo of," as shown below. This is expected to improve inference accuracy.

[0114] A movie / video of [X]_1 [X]_2,... [X]_M person wearing [Y]_1 [Y]_2+,... [Y]_M. In addition, in the above-described embodiment, an example is shown in which the image mask generation unit 33 performs mask processing to set areas with large gradients to 0, but this is not limited to this.

[0115] FIG. 7 is a diagram for explaining a modified example of the processing by the image mask generation unit 33 of the association device 1 according to an embodiment.

[0116] In FIG. 7, the symbol A indicates processing by the image mask generating unit 33 according to the embodiment described above, and the symbol B indicates processing by the image mask generating unit 33 according to the modified example.

[0117] As shown by symbol A, the image mask generation unit 33 of the above-described embodiment performs masking processing in which the top p% of areas with gradient values ​​are considered to be clothing information and set to 0, and other areas are considered to be other than clothing information and set to 1.

[0118] The image mask generation unit 33 may perform mask processing by weighting the regions other than the top p % of gradient values ​​according to the gradient value (strength). That is, the image mask generation unit 33 may set a value of 1 or less according to the gradient value (strength) for the regions other than the top p % of gradient values. This is expected to improve the inference accuracy.

[0119] In addition, although the above-described embodiment shows an example in which p=33.3, the present invention is not limited to this and p may be a value other than 33.3.

[0120] Furthermore, the above disclosure enables those skilled in the art to implement and manufacture the present embodiment.

[0121] REFERENCE NUMERALS 1 Corresponding device 2 Inference unit 21 Camera shooting unit 22 Frame image set storage unit 23 Image feature extraction unit 24 Person correlation unit 25 Person registration DB 26 Result output unit 3 Training unit 30 Machine learning model 31 Image input unit 32 Image feature extraction unit, image encoder 33 Image mask generation unit 34 Image mask processing unit 35 Text input unit 36 ​​Text feature extraction unit, text encoder 37 Text mask processing unit 38 Loss calculation unit 39 Training data DB 10 Computer 10a Processor 10b GPU 10c Memory 10d Storage unit 10e IF unit 10f IO unit 10g Reading unit 10h Program 10i Recording medium 10j Bus

Claims

1. An information processing device comprising: a control unit that executes processing to extract image features from image data; generate an image mask from the image features according to feature weights; generate masked image features by applying the image mask to the image features; generate masked text information by applying a text mask to additional text information that represents clothing of a person included in text information related to the content of the image data; extract masked text features from the masked text information; and calculate a loss due to estimation based on the masked image features and a loss due to estimation based on the masked text features.

2. The information processing apparatus according to claim 1, wherein said control unit executes a process of generating said additional text information based on label information related to said image data and adding said additional text information to said text information.

3. An information processing device comprising: a control unit that extracts image features from image data; generates an image mask from the image features according to feature weights; generates masked image features by applying the image mask to the image features; generates masked text information by applying a text mask to additional text information that represents clothing of a person included in text information related to the content of the image data; extracts masked text features from the masked text information; and executes processing to train a machine learning model based on the masked image features and the masked text features.

4. The information processing device described in claim 3, characterized in that the process of training the machine learning model includes a process of updating parameters of the machine learning model based on a loss due to estimation based on the masked image features and a loss due to estimation based on the masked text features.

5. The information processing device according to claim 3 or 4, characterized in that the control unit executes a process of generating the additional text information based on label information related to the image data and adding the additional text information to the text information.

6. An information processing program causing a computer to execute a process of extracting image features from image data, generating an image mask from the image features according to feature weights, generating masked image features by applying the image mask to the image features, generating masked text information by applying a text mask to additional text information indicating clothing of a person included in text information related to the content of the image data, extracting masked text features from the masked text information, and calculating a loss due to estimation based on the masked image features and a loss due to estimation based on the masked text features.

7. The information processing program according to claim 6, wherein the computer is caused to execute a process of generating the additional text information based on label information associated with the image data and adding the additional text information to the text information.

8. An information processing program that causes a computer to execute the following processes: extracting image features from image data; generating an image mask from the image features according to feature weights; generating masked image features by applying the image mask to the image features; generating masked text information by applying a text mask to additional text information that represents clothing of a person included in text information related to the content of the image data; extracting masked text features from the masked text information; and training a machine learning model based on the masked image features and the masked text features.

9. The information processing program according to claim 8, characterized in that the process of training the machine learning model includes a process of updating parameters of the machine learning model based on a loss due to estimation based on the masked image features and a loss due to estimation based on the masked text features.

10. The information processing program according to claim 8 or 9, which causes the computer to execute a process of generating the additional text information based on label information related to the image data and adding the additional text information to the text information.

11. An information processing method characterized by the steps of: extracting image features from image data; generating an image mask from the image features according to feature weights; generating masked image features by applying the image mask to the image features; generating masked text information by applying a text mask to additional text information indicating clothing of a person included in text information related to the content of the image data; extracting masked text features from the masked text information; and calculating a loss due to estimation based on the masked image features and a loss due to estimation based on the masked text features, wherein the computer executes the steps.

12. The information processing method according to claim 11, wherein the computer executes a process of generating the additional text information based on label information associated with the image data and adding the additional text information to the text information.

13. An information processing method, characterized in that a computer executes processes of extracting image features from image data, generating an image mask from the image features according to feature weights, generating masked image features by applying the image mask to the image features, generating masked text information by applying a text mask to additional text information indicating clothing of a person included in text information related to the content of the image data, extracting masked text features from the masked text information, and training a machine learning model based on the masked image features and the masked text features.

14. The information processing method described in claim 13, characterized in that the process of training the machine learning model includes a process of updating parameters of the machine learning model based on a loss due to estimation based on the masked image features and a loss due to estimation based on the masked text features.

15. The information processing method according to claim 13 or 14, wherein the computer executes a process of generating the additional text information based on label information associated with the image data and adding the additional text information to the text information.

Images