Computing device

Abstract

A computing device includes a storage device and a processor. The storage device is used for storing a program code, where the program code includes a contrastive learning based multimodal model. The processor is used to load and execute the program code, where the contrastive learning based multimodal model executes following inference operations: obtaining a plurality of first vectors corresponding to a plurality of first-type data, respectively; receiving an input data, where the input data includes a second-type data; generating a second vector corresponding to the second-type data; generating a plurality of output values according to the plurality of first vectors and the second vector; and referring to the plurality of output values to judge that multiple first-type data included in the plurality of first-type data are paired with the second-type data.

Claims

1. A computing device comprising: a storage device for storing code, wherein the code includes a contrastive learning-based multimodal model; and a processor for loading and executing the code, wherein the contrastive learning-based multimodal model performs the following inference operations: acquiring a plurality of first vectors respectively corresponding to a plurality of first type data; receiving input data, wherein the input data includes a second type data; generating a second vector corresponding to the input data; generating a plurality of output values ​​based on the plurality of first vectors and the second vector; and determining, based on the plurality of output values, that a plurality of the first type data among the plurality of first type data are simultaneously paired with the second type data.

2. The computing device as claimed in claim 1, wherein the contrastive learning-based multimodal model employs a contrastive language-image pre-trained model, wherein the plurality of first-type data are a plurality of different texts, and the second-type data is an image.

3. The computing apparatus as claimed in claim 2, wherein the contrastive learning-based multimodal model comprises: an image encoder for converting the image into an image vector; and a facial feature extraction model for extracting facial features from the image to generate a facial feature vector; the operation of generating the second vector corresponding to the input data comprises: merging the image vector and the facial feature vector to generate a merged vector; and generating the second vector based on the merged vector.

4. The computing device as described in claim 2, wherein the input data is an image captured by an in-vehicle monitoring device.

5. A computing device, comprising: a storage device for storing code, wherein the code includes a contrastive learning-based multimodal model; and a processor for loading and executing the code, wherein the contrastive learning-based multimodal model performs the following inference operations: acquiring a plurality of first vectors respectively corresponding to a plurality of first type data; receiving input data, wherein the input data includes a plurality of second type data; generating a second vector corresponding to the input data; generating a plurality of output values ​​based on the plurality of first vectors and the second vector; and determining, based on the plurality of output values, that one of the first type data among the plurality of first type data is paired with the input data.

6. The computing device as claimed in claim 5, wherein the contrastive learning-based multimodal model employs a contrastive language-image pre-training model, wherein the plurality of first-type data are a plurality of different images, and the plurality of second-type data are a plurality of different texts.

7. The computing apparatus of claim 6, wherein the multimodal model based on contrastive learning comprises: an image encoder for converting each of the plurality of different images into an image vector; and a facial feature extraction model for extracting facial features from each of the plurality of different images to generate a facial feature vector; the step of generating the plurality of first vectors respectively corresponding to the plurality of first type data comprises: merging the image vector and the facial feature vector to generate a merged vector; and generating a first vector among the plurality of first vectors based on the merged vector.

8. The computing device as described in claim 6, wherein the plurality of different images are images captured by an in-vehicle monitoring device, and the input data is input by a user.

9. A computing device comprising: a storage device for storing code, wherein the code includes a contrastive learning-based multimodal model, and the contrastive learning-based multimodal model includes a first type data encoder and a second type data encoder; and a processor for loading and executing the code, wherein the contrastive learning-based multimodal model performs the following training operations: receiving a plurality of first type training data and a plurality of second type training data; inputting the plurality of first type training data to the first type data encoder; inputting the plurality of second type training data to the second type data encoder; and training the parameters of the first and second type data encoders using a one-to-many pairing method, at least based on the outputs of the first and second type data encoders, such that the output of the contrastive learning-based multimodal model instructs one of the plurality of first type training data to be paired with a plurality of second type training data in the plurality of second type training data.

10. The computing apparatus of claim 9, wherein the contrastive learning-based multimodal model employs a contrastive language-image pre-training model, the first type of data encoder is an image encoder, the second type of data encoder is a text encoder, the plurality of first type of training data are a plurality of different images, and the plurality of second type of training data are a plurality of different texts.

11. The computing apparatus of claim 10, wherein the contrastive learning-based multimodal model further comprises: a pre-trained facial feature extraction model for extracting facial features from the plurality of different images; wherein the contrastive learning-based multimodal model trains parameters of the first and second type data encoders based on a combined result of the output of the first type data encoder and the output of the pre-trained facial feature extraction model, and the output of the second type data encoder.

Images