Information processing device, information processing method, and recording medium

Abstract

This information processing device comprises: a first processing means for outputting a first feature vector obtained by converting language related to an input image into a vector; a second processing means for outputting, on the basis of the first feature vector, a second feature vector obtained by converting a description of the input image into a vector; and a third processing means for outputting, on the basis of the second feature vector, the results of analyzing the input image.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] One known image analysis device is one that classifies sequential data using a classification index that indicates the likelihood of the class to which the sequential data belongs. For example, Patent Document 1 discloses a device that acquires multiple elements included in sequential data, calculates a classification index that indicates the likelihood of the class to which the sequential data belongs based on at least two of the multiple elements, and selects and outputs K classes (where K is a natural number equal to or less than N and is equal to or greater than 1) to which the sequential data is most likely to belong from N classes (where N is a natural number) that are candidates for classification of the sequential data based on the classification index.

[0003] International Publication No. 2022 / 113163

[0004] An object of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve the related techniques described above.

[0005] One aspect of the information processing device disclosed herein comprises a first processing means that outputs a first feature vector obtained by converting language related to an input image into a vector, a second processing means that outputs a second feature vector obtained by converting a description of the image into a vector based on the first feature vector, and a third processing means that outputs the results of an analysis of the image based on the second feature vector.

[0006] One aspect of the information processing method of this disclosure includes outputting a first feature vector obtained by converting language associated with an input image into a vector, outputting a second feature vector obtained by converting a description of the image into a vector based on the first feature vector, and outputting results of an analysis of the image based on the second feature vector, and is executed by a computer.

[0007] One aspect of the recording medium of this disclosure has recorded thereon a computer program that causes a computer to execute an information processing method including outputting a first feature vector obtained by converting language associated with an input image into a vector, outputting a second feature vector obtained by converting a description of the image into a vector based on the first feature vector, and outputting the results of an analysis of the image based on the second feature vector.

[0008] FIG. 1 is a block diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 2 is a flowchart showing an example of a flow of information processing operations in an information processing device according to the present disclosure. FIG. 3 is a block diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 4 is a flowchart showing an example of a flow of information processing operations in an information processing device according to the present disclosure. FIG. 5 is a block diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 6 is a flowchart showing an example of a flow of information processing operations in an information processing device according to the present disclosure. FIG. 7 is a block diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 8 is a flowchart showing an example of a flow of information processing operations in an information processing device according to the present disclosure. FIG. 9 is a block diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 10 is a flowchart showing an example of a flow of information processing operations in an information processing device according to the present disclosure.

[0009] Hereinafter, embodiments of an information processing device, an information processing method, and a recording medium will be described with reference to the drawings. [1: First Embodiment]

[0010] A first embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 1 and Fig. 2. In the following, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 10.

[0011] 1, the information processing device 10 includes a first processing unit 11, a second processing unit 12, and a third processing unit 13. The operation of the information processing device 10 will be described with reference to the flowchart of FIG.

[0012] 2, the first processing unit 11 outputs information (referred to as a "first feature vector") obtained by converting language related to the input image into a vector (step S11). For example, the first processing unit 11 may use a vision and language model (VLM) to project the input image into a common space for images and language, and convert the input image into the first feature vector.

[0013] The second processing unit 12 outputs information (referred to as a "second feature vector") obtained by converting a description of the input image into a vector based on the first feature vector (step S12). The second feature vector may also be referred to as information obtained by converting a sentence describing the input image into a vector. The second feature vector may also be referred to as information obtained by converting an interpretation of language related to the input image into a vector. The second processing unit 12 may output the second feature vector in response to the input of the first feature vector using a large language model (LLM).

[0014] The third processing unit 13 outputs an analysis result of the input image based on the second feature vector (step S13). In this embodiment, each of the operations according to the predetermined purpose may be collectively referred to as an "analysis." In other words, the third processing unit 13 may perform an operation according to the predetermined purpose based on the second feature vector and output the operation result.

[0015] For example, analyzing the image may include detecting a predetermined object in the image, i.e., the predetermined objective may include detecting a predetermined object in the input image, or analyzing the input image may include classifying the input image, i.e., the predetermined objective may include classifying the input image.

[0016] In other words, the third processing unit 13 analyzes the input image based on the input image description. In other words, the third processing unit 13 outputs an operation result according to a predetermined purpose when the image description is input.

[0017] For example, when it is desired to detect a predetermined object from an image, the third processing unit 13 may output, based on the description of the image, an indicator indicating the possibility of the presence of the predetermined object in the image, information indicating the location in the image where the predetermined object is likely to be present, etc. Furthermore, when it is desired to classify objects included in an image as living or non-living, the third processing unit 13 may output, based on the description of the image, an indicator indicating the likelihood of a "living object," an indicator indicating the likelihood of a "non-living object," etc.

[0018] In this way, the information processing device 10 performs an information processing method that includes outputting a first feature vector obtained by converting language related to an input image into a vector, outputting a second feature vector obtained by converting a description of the input image into a vector based on the first feature vector, and outputting an analysis result of the input image based on the second feature vector.

[0019] The information processing device 10 described above may be realized by a computer reading a computer program recorded on a recording medium. In this case, the computer program may cause the computer to execute an information processing method including: outputting a first feature vector obtained by converting language associated with an input image into a vector; outputting a second feature vector obtained by converting a description of the input image into a vector based on the first feature vector; and outputting an analysis result of the input image based on the second feature vector. [Technical Effect]

[0020] In the information processing device 10 disclosed herein, the first processing unit 11 and the second processing unit 12 output a description of the input image, and the third processing unit 13 analyzes the input image based on the description of the input image. Since the first processing unit 11 and the second processing unit 12 output a description of the time-series image group, it is possible to explain the basis of the analysis result output by the third processing unit 13. Since the third processing unit 13 analyzes the input image based on the description of the time-series image group, it is possible to improve the accuracy of the analysis. [2: Second Embodiment]

[0021] A second embodiment of the information processing device, information processing method, and recording medium will be described with reference to FIGS. 3 to 6. Hereinafter, the second embodiment of the information processing device, information processing method, and recording medium will be described using an information processing device 20. Note that, for the second embodiment, descriptions that overlap with the description of the first embodiment will be omitted as appropriate. [2-1: Configuration of the information processing device 20]

[0022] The configuration of the information processing device 20 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 20.

[0023] 3 , the information processing device 20 includes a calculation device 21 and a storage device 22. The information processing device 20 may further include a communication device 23, an input device 24, and an output device 25. However, the information processing device 20 does not necessarily include at least one of the communication device 23, the input device 24, and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0024] The arithmetic device 21 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit).

[0025] The arithmetic device 21 reads a computer program 221 including at least one of computer program code and computer program instructions. For example, the arithmetic device 21 may read the computer program 221 stored in the storage device 22. For example, the arithmetic device 21 may read the computer program 221 stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the information processing device 20. The computer program 221 read from the recording medium may be stored in the storage device 22. The arithmetic device 21 may acquire (i.e., download or read) the computer program 221 from a device (not shown) located outside the information processing device 20 via the communication device 23 (or another communication device). The downloaded computer program 221 may be stored in the storage device 22.

[0026] The arithmetic device 21 executes the loaded computer program 221. As a result, logical functional blocks for executing information processing to be performed by the information processing device 20 are realized within the arithmetic device 21. In other words, the arithmetic device 21, together with the storage device 22 or the like in which the computer program 221 is recorded (in other words, together with the storage device 22 and the computer program 221 recorded in the storage device 22 or the like), can function as a controller or computer for realizing logical functional blocks for executing processing to be performed by the information processing device 20. In other words, the at least one processor included in the arithmetic device 21, the memory (recording medium) included in the storage device 22 or the like, and the computer program 221 are configured so that the information processing device 20 performs the information processing to be performed by the information processing device 20.

[0027] A computational model that can be constructed by machine learning may be implemented in the computational device 21 by the computational device executing the computer program 221. An example of a computational model that can be constructed by machine learning is a computational model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the computational model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The computational device 21 may execute a first feature vector output process, a second feature vector output process, and an image analysis process using the computational model. That is, the operations of executing the first feature vector output process, the second feature vector output process, and the image analysis process may include operations of executing the first feature vector output process, the second feature vector output process, and the image analysis process using the computational model. Note that a computational model that has been constructed by offline machine learning using training data may be implemented in the computational device 21. Furthermore, the computational model implemented in the computational device 21 may be updated by online machine learning on the computational device 21. Alternatively, the calculation device 21 may perform the first feature vector output process, the second feature vector output process, and the image analysis process using a calculation model implemented in a device external to the calculation device 21 (i.e., a device provided outside the information processing device 20) in addition to or instead of the calculation model implemented in the calculation device 21.

[0028] The recording medium for recording the computer program 221 executed by the arithmetic device 21 may be at least one of a CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, Blu-ray (registered trademark), or other optical disk, a magnetic medium such as a magnetic tape, a magneto-optical disk, a semiconductor memory such as a USB memory, or any other medium capable of storing a program. The recording medium may include a device capable of recording a computer program (for example, a general-purpose device or a dedicated device in which the computer program 221 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 221 may be realized by a logical processing block realized within the arithmetic device 21 when the arithmetic device 21 (i.e., processor) executes the computer program 221, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) provided in the arithmetic device 21, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0029] The storage device 22 includes at least one memory capable of storing desired data. In other words, the storage device 22 includes at least one memory containing desired data. For example, the storage device 22 may store a computer program 221 executed by the arithmetic device 21. In this case, the storage device 22 (memory) may be used as the above-mentioned recording medium for recording the computer program 221 executed by the arithmetic device 21. The storage device 22 may temporarily store data used by the arithmetic device 21 when the arithmetic device 21 is executing the computer program 221. The storage device 22 may also store data to be stored long-term by the information processing device 20. The storage device 22 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 22 may include a non-temporary recording medium.

[0030] The communication device 23 may be capable of communicating with devices external to the information processing device 20. The communication device 23 may perform wired communication or wireless communication.

[0031] The input device 24 is a device capable of accepting information input to the information processing device 20 from outside. The input device 24 may include an operation device (e.g., a keyboard, a mouse, a touch panel, etc.) that can be operated by a user of the information processing device 20. The input device 24 may include a recording medium reading device that can read information recorded on a recording medium that is detachable from the information processing device 20, such as a USB (Universal Serial Bus) memory. Note that when information is input to the information processing device 20 via the communication device 23 (in other words, when the information processing device 20 acquires information via the communication device 23), the communication device 23 may function as an input device.

[0032] The output device 25 is a device capable of outputting information to the outside of the information processing device 20. The output device 25 may output visual information such as text or images, auditory information such as sound, or tactile information such as vibration, as the information. The output device 25 may include, for example, at least one of a display, a speaker, a printer, and a vibration motor. The output device 25 may be capable of outputting information to a recording medium detachable from the information processing device 20, such as a USB memory. Note that when the information processing device 20 outputs information via the communication device 23, the communication device 23 may function as the output device.

[0033] 3 shows an example of logical functional blocks implemented in the arithmetic device 21 to perform speech recognition. As shown in FIG. 3, a first processing unit 211, a second processing unit 212, a third processing unit 213, and an image acquisition unit 214 may be implemented in the arithmetic device 21. The "first processing unit 211" is a component corresponding to the "first processing unit 11" in the first embodiment described above, the "second processing unit 212" is a component corresponding to the "second processing unit 12" in the first embodiment described above, and the "third processing unit 213" is a component corresponding to the "third processing unit 13" in the first embodiment described above. [2-2: Information Processing Method Executed by Information Processing Device 20]

[0034] The information processing device 20 is configured as a device for analyzing a time-series image group. The time-series image group may be a group of images constituting a video. For example, the information processing device 20 may be configured as a device that sequentially acquires images included in a video and classifies the types of objects included in the images. The time-series image group may also be a group of images of an object captured at predetermined intervals. For example, the time-series image group may be a group of images of an object captured daily, weekly, or monthly. For example, the information processing device 20 may be configured as a device that sequentially acquires images included in an image group and predicts the future. For example, the information processing device 20 may predict the probability of precipitation based on a time-series satellite image group. For example, the information processing device 20 may predict a lesion based on a time-series endoscopic examination image group.

[0035] An information processing method executed by the information processing device 20 will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart showing an example of the flow of the information processing method executed by the information processing device 20. Fig. 5 is a block diagram showing an example of the flow of information in the information processing method executed by the information processing device 20.

[0036] 4, the image acquisition unit 214 acquires images (step S21). The images acquired by the image acquisition unit 214 are images included in a time-series image group. The image acquisition unit 214 acquires images in time-series order.

[0037] Note that when the information processing operation by the information processing device 20 starts, the current time t may be set to 0. In the information processing method described with reference to Figures 4 and 5, the current time t may be used as a counter for counting the number of times an image has been acquired, and the current time t may be incremented each time an image is acquired. In other words, the current time t may indicate the number of times the current acquisition has occurred.

[0038] The first processing unit 211 converts the language associated with the acquired image into a vector and outputs the current first feature vector (step S22). The first processing unit 211 may output the first feature vector using a VLM. The VLM used by the first processing unit 211 may be pre-trained.

[0039] The second processing unit 212 outputs a second feature vector based on the first feature vector and the previously output second feature vector (step S23). The second processing unit 212 may output the second feature vector using an LLM. The LLM used by the second processing unit 212 may be pre-trained. The first processing unit 211 and the second processing unit 212 may be pre-trained and configured to generate a description of an input image regardless of the domain of the input image.

[0040] The second feature vector may be an internal feature of the second processing unit 212. In other words, the second processing unit 212 sequentially updates the internal feature based on the sequentially input first feature vector. The second processing unit 212 may output a current second feature vector based on the current first feature vector and the second feature vector output previously. As illustrated in FIG. 5 , the second processing unit 212 may receive the current first feature vector (t) and the previously output second feature vector (t-1) and output the current second feature vector (t). Furthermore, at the next time (t+1), the second processing unit 212 may receive the next first feature vector (t+1) and the current second feature vector (t) and output the next second feature vector (t+1). Because the current second feature vector (t) is based on the previously output second feature vector (t-1), it can be said that the second feature vector contains information contained in the previously output second feature vector. Furthermore, it can be said that the second feature vector contains information contained in the first feature vector that was output previously.

[0041] A new first feature vector is input to the second processing unit 212 each time an image is acquired. In other words, at the current time t, the second processing unit 212 is input with the first feature vectors output by the first processing unit 211 up to the current time t. In other words, the second processing unit 212 receives the first feature vectors sequentially, integrates the input information, and outputs the integrated information. The second processing unit 212 converts the description of the time-series image group into a vector and outputs the second feature vector for analysis by the third processing unit 213.

[0042] The third processing unit 213 outputs the results of analyzing the time-series image group based on the current second feature vector (step S24). As illustrated in FIG. 5, the second processing unit 212 may receive the current second feature vector (t) and output the current analysis result (t). In other words, the third processing unit 213 outputs the analysis result of the time-series image group acquired up to this point. Note that, as illustrated in FIG. 5, at the next time (t+1), the second processing unit 212 may receive the next second feature vector (t+1) and output the next analysis result (t+1).

[0043] The first processing unit 211, the second processing unit 212, and the third processing unit 213 may include a neural network. The third processing unit 213 may correspond to a fully connected layer. The third processing unit 213 may output information summarizing the information processed by the first processing unit 211 and the second processing unit 212. The third processing unit 213 may be trained to adapt to an action according to a predetermined purpose. The third processing unit 213 may be trained to perform an action according to a predetermined purpose based on the second feature vector. In other words, the third processing unit 213 may be constructed to perform an action according to a predetermined purpose based on a sentence. The third processing unit 213 may be trained using a loss function. The third processing unit 213 may be trained using, for example, softmax cross entropy.

[0044] The third processing unit 213 determines whether to end the analysis of the time-series image group (step S25). That is, the third processing unit 213 determines whether the image acquisition unit 214 will acquire the next image or whether acquisition will end with the current image. The third processing unit 213 may determine to end the processing, for example, when all images included in the time-series image group have been processed. The third processing unit 213 may also determine to end the sequential processing (the processing from step S21 to step S24) when the analysis result can be confirmed. Note that the operation of determining the end of the sequential processing may be included in the operation according to a predetermined purpose.

[0045] If it is determined that the analysis of the time-series image group is to be ended (step S25: Yes), the information processing operation ends. If it is determined that the analysis of the time-series image group is not to be ended (step S25: No), the process returns to step S21. [2-3: Specific Example]

[0046] A specific example will be given below: As a specific example, a case will be described in which the third processing unit 213 operates in accordance with the purpose of determining whether or not the face of a person included in an image is recognizable.

[0047] For example, consider a case where, at t=1, the second processing unit 212 outputs a second feature vector obtained by converting a sentence such as "A person wearing a hoodie is walking. The surroundings are dark" based on a first feature vector obtained by converting language related to an acquired image. In this case, the third processing unit 213 may output a determination result that the face of the person included in the image cannot be recognized.

[0048] For example, at t=2, the second processing unit 212 may generate a description of a time-series group of images based on language associated with the images acquired at t=2 and the description of the images acquired at t=1. For example, consider a case where the second processing unit 212 outputs a second feature vector obtained by converting a sentence, "The same person is approaching. He is wearing glasses and operating a mobile device," based on a first feature vector obtained by converting language associated with the images acquired at t=2 and the second feature vector output at t=1. In this case, the third processing unit 213 may output a determination result that the face of the person included in the image cannot be recognized.

[0049] For example, consider the case where, at t=3, the second processing unit 212 outputs a second feature vector obtained by converting the sentence "A person's face has been confirmed" based on the first feature vector obtained by converting the language related to the image acquired at t=3 and the second feature vector output at t=2. In this case, the third processing unit 213 may output a determination result that the face of the person included in the image is recognizable. Furthermore, since the person's face can now be confirmed, the third processing unit 213 may decide to end the analysis of the time-series image group at t=3. In other words, the sequential processing may end when a predetermined purpose is achieved. In the case where the predetermined purpose is to recognize a face as described above, the sequential processing may end when an image in which the face is recognizable has been acquired. [2-4: Modification]

[0050] 6, when the third processing unit 213 determines to end the analysis of the time-series image group (step S26: Yes), it may output the results of the analysis of the time-series image group based on the second feature vector (step S27). In other words, until it is determined to end the analysis of the time-series image group, the third processing unit 213 may only determine whether or not to end the analysis of the time-series image group. The third processing unit 213 may output the analysis results of the time-series image group based on the second feature vectors output by the second processing unit 212 up to this point. [2-5: Specific Example 1 of Modified Example]

[0051] The modified examples will be described below with reference to specific examples. As specific example 1, a case will be described in which the third processing unit 213 operates in accordance with the purpose of determining what kind of video a time-series group of images belongs to. The first processing unit 211 sequentially acquires images included in the group of images constituting the video, and sequentially outputs first feature vectors. The second processing unit 212 outputs the current second feature vector based on the first feature vector and the second feature vector output previously.

[0052] In specific example 1, in the early part of the video, the second processing unit 212 may output a second feature vector converted from a sentence, "A person holding a photo in front of his face is visible." In addition, in the middle of the video, the second processing unit 212 may output a second feature vector converted from a sentence, "A person holding a photo in front of his face is approaching."

[0053] When all images included in the group of images constituting the moving image have been processed, the third processing unit 213 may output a determination result. The third processing unit 213 may output a determination result that the moving image is a "spoofed (SPOOF)" moving image based on the second feature vector described above. [2-6: Specific Example 2 of Modification]

[0054] As a specific example 2, a case will be described in which the third processing unit 213 operates in accordance with the purpose of detecting sections during a race from a video, which is a group of time-series images.

[0055] In specific example 2, in the early part of the video, the second processing unit 212 may output a second feature vector obtained by converting a sentence such as "An athlete is on screen." In addition, in the middle of the video, the second processing unit 212 may output a second feature vector obtained by converting a sentence such as "An athlete has started running. It looks like a track and field event." In addition, in the middle of the video, the second processing unit 212 may output a second feature vector obtained by converting a sentence such as "This is a long jump event."

[0056] When all images included in the group of images constituting the video have been processed, the third processing unit 213 may output a discrimination result. Based on the second feature vector described above, the third processing unit 213 may output a detection result that the section in the video from when the athlete starts running until when the athlete lands is the long jump competition section. [2-7: Technical Effects]

[0057] In the information processing device 20 disclosed herein, the first processing unit 211 and the second processing unit 212 output a description of the time-series image group, and the third processing unit 213 analyzes the input image based on the description of the time-series image group. Since the first processing unit 211 and the second processing unit 212 output a description of the time-series image group, it is possible to explain the basis for the analysis results output by the third processing unit 213. Since the third processing unit 213 analyzes the input image based on the description of the time-series image group, it is possible to improve the accuracy of the analysis. [3: Third Embodiment]

[0058] A third embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Figures 7 and 8. Hereinafter, the third embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 30. Note that, for the third embodiment, descriptions that overlap with the descriptions of the first and second embodiments will be omitted as appropriate. Note that, in the drawings, parts common to the first and second embodiments are designated by the same reference numerals.

[0059] 7, the arithmetic device 21 included in the information processing device 30 includes, as logical functional blocks, a first processing unit 211, a second processing unit 312, a third processing unit 213, and an image acquisition unit 214. The second processing unit 312 may include an explanation receiving unit 3121. [3-1: Information Processing Method Executed by the Information Processing Device 30]

[0060] An information processing method executed by the information processing device 30 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the flow of the information processing method executed by the information processing device 30.

[0061] 8, the explanation receiving unit 3121 receives information (referred to as "explanation information") that explains the operation of the third processing unit 213 (step S31). The explanation information may be text. In other words, the explanation receiving unit 3121 receives instructions for the second processing unit 312 so that the second processing unit 312 outputs information suitable for the operation of the third processing unit 213. The second processing unit 312, which has only undergone pre-learning, does not know the operation performed by the third processing unit 213. For this reason, the explanation receiving unit 3121 receives input of the explanation information at the beginning of the information processing operation.

[0062] For example, the explanatory information (instructions to the second processing unit 312) may be, "We will now perform a facial impersonation detection task. Please answer whether the face in the image is a real face, a face impersonated using a photograph or display, or whether you cannot distinguish." The second processing unit 312 outputs a second feature vector based on the instructions. For example, in the early part of the video, the second processing unit 312 may output a second feature vector converted from the sentence, "The lighting conditions are poor, so I can't distinguish." Also, in the middle of the video, the second processing unit 312 may output a second feature vector converted from the sentence, "This is a real person. It doesn't seem to be an impersonation." In other words, in this example, the image description generated by the second processing unit 312 may be related to facial discrimination. In this way, when instructions are given, the second processing unit 312 can output more useful information for discrimination compared to the specific example described above. In other words, the second processing unit 312, using a pre-trained LLM, can generate an image description tailored to a specific purpose by receiving explanatory information. [3-2: Technical Effects]

[0063] The information processing device 30 according to this disclosure can generate an image description according to a predetermined purpose by having the second processing unit 312 receive the description information. The second processing unit 312 can generate an image description according to a predetermined purpose by receiving the description information without performing learning according to the predetermined purpose. [4: Fourth Embodiment]

[0064] A fourth embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Figures 9 and 10. Hereinafter, the fourth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 40. Note that, for the fourth embodiment, descriptions that overlap with the descriptions of the first to third embodiments will be omitted as appropriate. Note that, in the drawings, parts common to the first to third embodiments are designated by the same reference numerals.

[0065] As shown in Fig. 9, the arithmetic device 21 included in the information processing device 40 includes, as logical functional blocks, a first processing unit 211, a second processing unit 412, a third processing unit 413, and an image acquisition unit 214. The third processing unit 413 in the fourth embodiment determines to end the sequential processing when a predetermined purpose is achieved. Furthermore, when determining to end the sequential processing, the third processing unit 413 outputs an instruction to the second processing unit 412 to summarize the basis for the determination. [4-1: Information Processing Method Executed by the Information Processing Device 40]

[0066] An information processing method executed by the information processing device 40 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the flow of the information processing method executed by the information processing device 40.

[0067] As shown in FIG. 10 , the image acquisition unit 214 acquires an image (step S21). The first processing unit 211 converts language associated with the acquired image into a vector and outputs a current first feature vector (step S22). The second processing unit 412 outputs a second feature vector based on the first feature vector and the previously output second feature vector (step S23). The third processing unit 413 determines whether to end the analysis of the time-series image group (step S26). If the third processing unit 413 determines to end the analysis of the time-series image group (step S26: Yes), the second processing unit 412 outputs a summary of the analysis based on the previously output second feature vector (step S41).

[0068] For example, the third processing unit 413 may output an instruction to the second processing unit 412 saying, "A final decision has been made. Please summarize the analysis results so far." In this case, the second processing unit 412 may output a sentence expressed in natural language such as, "In the first half of the frames, the lighting conditions were poor and the face could not be confirmed. Also, the face was difficult to determine because the person was wearing glasses and a hood. However, as the person gradually got closer, the face could be confirmed."

[0069] In the fourth embodiment, similarly to the third embodiment, instructions to the second processing unit 412 may be accepted. [4-2: Technical Effects]

[0070] The second processing unit 412 can output a sentence expressing the second feature vector in natural language each time a first feature vector is input. However, even if the second processing unit 412 were to output an explanation for every image included in a time-series image group (for example, a group of images constituting a video), it would be unrealistic to check the explanation for every image. Therefore, the information processing device 40 according to this disclosure outputs the judgment criteria collectively when a final judgment is made. The information processing device 40 can present the basis of the analysis result in natural language. [5: Fifth Embodiment]

[0071] A fifth embodiment relating to an information processing device, an information processing method, and a recording medium will be described with reference to Figures 11 to 13. Hereinafter, the fifth embodiment relating to an information processing device, an information processing method, and a recording medium will be described using an information processing device 50. Note that, for the fifth embodiment, descriptions that overlap with the descriptions of the first to fourth embodiments will be omitted as appropriate. Note that, in the drawings, parts common to the first to fourth embodiments are designated by the same reference numerals.

[0072] 11 , the arithmetic device 21 included in the information processing device 50 includes, as logical functional blocks, a first processing unit 211, a second processing unit 512, a third processing unit 513, and an image acquisition unit 214. The third processing unit 513 may include an estimation unit 5131, a determination unit 5132, and a classification unit 5133. Each of the estimation unit 5131, the determination unit 5132, and the classification unit 5133 may be trained to operate according to its respective purpose based on the second feature vector output by the second processing unit 512, which is successively updated.

[0073] The estimation unit 5131 operates in accordance with the purpose of estimating an index indicating to which of multiple classes a group of time-series images belongs. The estimation unit 5131 may be trained using a loss function for estimating the index.

[0074] The decision unit 5132 performs an operation according to the purpose of determining whether to terminate or continue the sequential processing including the image acquisition processing and the image description generation processing. In other words, the decision unit 5132 performs an operation according to the purpose of determining whether to make a final decision or to reserve the final decision.

[0075] The classification unit 5133 operates according to the purpose of classifying a group of time-series images into one of a plurality of classes. The classification unit 5133 may operate when the determination unit 5132 determines to make a final determination.

[0076] In this embodiment, the estimation unit 5131 and the determination unit 5132 may operate the same number of times as the number of images acquired. In contrast, the classification unit 5133 may operate when the determination unit 5132 determines to make a final determination. [5-1: Information Processing Method Executed by the Information Processing Device 50]

[0077] The information processing device 50 according to the fifth embodiment is configured as a device for classifying a group of time-series images. In other words, the information processing device 50 is configured as a device for solving a classification problem of sequential data.

[0078] 12 and 13, an information processing method executed by the information processing device 50 will be described. Fig. 12 is a flowchart showing an example of the flow of the information processing method executed by the information processing device 50. Fig. 13 is a block diagram showing an example of the flow of information in the information processing method executed by the information processing device 50.

[0079] 12 , the image acquisition unit 214 acquires images (step S21). The image acquisition unit 214 sequentially acquires images included in the time-series image group in chronological order. For example, the image acquisition unit 214 acquires the images included in the time-series image group in order for each acquisition unit. For example, if the time-series image group is a group of images that constitute a video, the image acquisition unit 214 may acquire one frame at a time in order, or may acquire multiple frames at a time in order. The acquisition unit may be determined arbitrarily depending on the requirements for applying the classification operation.

[0080] The image acquisition unit 214 may acquire data directly from any data acquisition device (e.g., a camera, a microphone, etc.). Alternatively, the image acquisition unit 214 may read data that has been acquired in advance by a data acquisition device and stored in storage, etc. When acquiring data from a camera, the image acquisition unit 214 may be configured to acquire data from each of a plurality of cameras. The first processing unit 211 outputs the current first feature vector obtained by converting language related to the acquired image into a vector (step S22).

[0081] The second processing unit 512 outputs a first second feature vector based on the first feature vector and the second feature vector previously output (step S51). In the fifth embodiment, the second feature vector output by the second processing unit 512 based on the first feature vector is referred to as the first second feature vector.

[0082] The estimation unit 5131 estimates an index based on the first second feature vector (step S52). The estimation unit 5131 may estimate at least one of a likelihood ratio and a posterior probability as the "index." In this embodiment, a case will be described in which the estimation unit 5131 operates in accordance with the purpose of estimating a likelihood ratio.

[0083] Here, the "likelihood ratio" is an example of an index indicating to which of multiple candidate classes a group of time-series images belongs. The "likelihood ratio" indicates the likelihood that a group of time-series images belongs to a certain class among multiple classes. The estimation unit 5131 may estimate a likelihood ratio corresponding to each of the multiple classes.

[0084] The likelihood ratio may be a value that can be obtained as a result of likelihood ratio learning using a SPRT-based algorithm that treats sequence data as an Nth-order Markov process (SPRT-TANDEM).

[0085] The estimation unit 5131 may estimate the logarithm of the likelihood ratio (Log Likelihood Ratio (LLR)). In the following description, the log likelihood ratio may be simply referred to as the likelihood ratio.

[0086] The estimation unit 5131 converts a sentence representing the estimated likelihood ratio into a vector and outputs the third feature vector to the second processing unit 512 (step S53). In other words, the estimation unit 5131 feeds back the estimation result based on the output of the second processing unit 512 to the second processing unit 512. The second processing unit 512 receives the estimation result as feedback and outputs second second feature information converted from an explanation of the estimation result into a vector (step S54). In the fifth embodiment, the second feature vector output by the second processing unit 512 based on the third feature vector is referred to as the second second feature vector.

[0087] When the third feature vector is input, the second processing unit 512 may output a second feature vector obtained by converting a sentence indicating at least one of the likelihood that the sequential processing should be terminated and the likelihood that the sequential processing should be continued. In this disclosure, termination may refer to the action of classifying using the likelihood ratio estimated at present. Furthermore, continuation in this disclosure may refer to the action of acquiring further images and estimating likelihood ratios without classifying using the likelihood ratio estimated at present.

[0088] The decision unit 5132 decides whether to terminate or continue the sequential processing (step S55). The decision unit 5132 decides whether to terminate or continue and outputs the result of the decision. In other words, the decision unit 5132 decides whether to make a termination decision or to postpone the termination decision. In other words, the decision unit 5132 decides whether to classify the time-series image group into one of a plurality of classes based on the image group acquired up to this point.

[0089] If it is determined that the sequential processing is to be terminated (step S55: end), the classification unit 5133 classifies the group of time-series images into one of a plurality of classes (step S56). The classification unit 5133 classifies the group of time-series images into one of a plurality of classes and outputs the classification result. If it is determined that the sequential processing is to be continued (step S55: continue), the process returns to step S21. That is, the information processing device 50 repeats the sequential processing until the determination unit 5132 determines that the sequential processing is to be terminated.

[0090] The classification unit 5133 may classify the group of time-series images into the class corresponding to the largest likelihood ratio among the plurality of classes. For example, when classifying the group of time-series images into one of two classes, class 0 and class 1, if the likelihood ratio corresponding to class 0 is larger than the likelihood ratio corresponding to class 1, the classification unit 5133 classifies the group of time-series images into class 0. Similarly, if the likelihood ratio corresponding to class 1 is larger than the likelihood ratio corresponding to class 0, the classification unit 5133 classifies the group of time-series images into class 1. Note that the plurality of classes may be three or more.

[0091] In this way, the information processing device 50 performs sequential operations to classify a group of time-series images into appropriate classes. The estimation unit 5131 may be configured to estimate a larger likelihood ratio for the class to be classified as the number of acquired images increases.

[0092] 13 illustrates an example in which a time-series group of images is classified into two classes: a genuine class and a spoof class. As illustrated in FIG. 13, at t=1, the image acquisition unit 214 acquires an image (t=1). The first processing unit 211 outputs a first vector v1 (t=1). The second processing unit 512 outputs a first second vector v21 (t=1).

[0093] Consider a case where the estimation unit 5131 estimates the likelihood ratio to be "0.5." In this case, the estimation unit 5131 converts the estimation result, "The likelihood ratio of the genuine class / impersonation class is 0.5," into a vector and outputs the third feature information v3 (t=1) to the second processing unit 512. The second processing unit 512 receives the estimation result as feedback and outputs a second second feature vector v22 (t=1) that converts the sentence, "The likelihood ratio of the genuine class / impersonation class is 0.5, which is not high, so classification is not yet possible." Based on the image group acquired up to this point, the decision unit 5132 decides not to classify the time-series image group into any of the multiple classes and reserves the decision to terminate.

[0094] Next, at t=2, the image acquisition unit 214 acquires an image (t=2). The first processing unit 211 outputs a first vector v1 (t=2). The second processing unit 512 outputs a first second vector v21 (t=2).

[0095] Consider a case where the estimation unit 5131 estimates the likelihood ratio to be "15.2." In this case, the estimation unit 5131 converts the estimation result, "The likelihood ratio of the genuine class / impersonation class is 15.2," into a vector and outputs the third feature information v3 (t=2) to the second processing unit 512. The second processing unit 512 receives the estimation result as feedback and outputs a second second feature vector v22 (t=2) that converts the sentence, "The likelihood ratio of the genuine class / impersonation class is 15.2, which is sufficiently high, and it appears that it can be safely classified into the genuine class." The determination unit 5132 determines to classify the time-series image group based on the image group acquired up to this time, and determines to end the process. The classification unit 5133 outputs a classification result (t=2) that the time-series image group belongs to the genuine class.

[0096] In the fifth embodiment, as in the third embodiment, instructions to the second processing unit 412 may be accepted. Also, in the fifth embodiment, as in the fourth embodiment, a summary of the classification operation may be output. [5-2: Technical Effects]

[0097] The information processing device 50 according to this disclosure estimates indices and feeds back sentences explaining the indices to the second processing unit 512, thereby enabling accurate classification of a time-series image group. Since the estimation of indices by the estimation unit 5131 is based on a description of the time-series image group, it is possible to explain the basis for estimating the indices.

[0098] [6: Supplementary Note] The above-described embodiment can be further described as in the following supplementary note, but is not limited to the following. [Supplementary Note 1] An information processing device comprising: a first processing means that outputs a first feature vector obtained by converting language associated with an input image into a vector; a second processing means that outputs a second feature vector obtained by converting a description of the image into a vector based on the first feature vector; and a third processing means that outputs a result of analysis of the image based on the second feature vector. [Supplementary Note 2] The information processing device according to Supplementary Note 1, comprising: acquisition means that acquires the images included in a time-series image group in chronological order; the first processing means outputs the current first feature vector obtained by converting language associated with the acquired image into a vector; the second processing means outputs the current second feature vector based on the current first feature vector and the second feature vector output previously; and the third processing means outputs the result of analysis of the time-series image group based on the current second feature vector. [Supplementary Note 3] The information processing device according to Supplementary Note 1 or 2, wherein the second processing means receives explanatory information explaining the operation performed by the third processing means, and outputs the second feature vector based on the explanatory information. [Supplementary Note 4] The information processing device according to Supplementary Note 2, wherein the third processing means decides whether or not to make the analysis based on the current second feature vector the final analysis. [Supplementary Note 5] The information processing device according to Supplementary Note 4, wherein, if it is decided that the analysis based on the current second feature vector is the final analysis, the second processing means outputs a summary of the analysis based on the second feature vector output previously.[Supplementary Note 6] The information processing device according to Supplementary Note 2, wherein the third processing means estimates an index indicating to which of a plurality of classes the group of time-series images belongs, based on the second feature vector based on the first feature vector, the second processing means converts an explanation regarding classifying the group of time-series images into one of the plurality of classes, based on the index, and outputs a second feature vector obtained by converting into a vector an explanation regarding classifying the group of time-series images into one of the plurality of classes, and the third processing means determines whether to terminate or continue sequential processing including at least acquisition processing by the acquisition means, and if it is determined to terminate the sequential processing, classifies the group of time-series images into one of the plurality of classes. [Supplementary Note 7] The information processing device according to Supplementary Note 1, wherein the analysis of the images includes at least one of classifying the group of time-series images and detecting a predetermined object in the images. [Supplementary Note 8] The information processing device according to Supplementary Note 2, wherein the analysis of the group of time-series images includes at least one of classifying the group of time-series images, detecting a predetermined object in the group of time-series images, future prediction based on the group of time-series images, and likelihood ratio estimation based on the group of time-series images. [Supplementary Note 9] The information processing device according to Supplementary Note 6, wherein the index is a likelihood ratio indicating the likelihood that the group of images in the time series belong to a certain class of the plurality of classes. [Supplementary Note 10] An information processing method executed by a computer, comprising: outputting a first feature vector obtained by converting language associated with an input image into a vector; outputting a second feature vector obtained by converting a description of the image into a vector based on the first feature vector; and outputting a result of analysis of the image based on the second feature vector. [Supplementary Note 11] A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method, comprising: outputting a first feature vector obtained by converting language associated with an input image into a vector; outputting a second feature vector obtained by converting a description of the image into a vector based on the first feature vector; and outputting a result of analysis of the image based on the second feature vector.

[0099] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical idea of ​​this disclosure.

[0100] 10, 20, 30, 40, 50 Information processing device 11, 211 First processing unit 12, 212, 312, 412, 512 Second processing unit 13, 213, 413, 513 Third processing unit 214 Image acquisition unit 3121 Explanation reception unit 5131 Estimation unit 5132 Determination unit 5133 Classification unit

Claims

1. An information processing device comprising: a first processing means for outputting a first feature vector obtained by converting language associated with an input image into a vector; a second processing means for outputting a second feature vector obtained by converting a description of the image into a vector based on the first feature vector; and a third processing means for outputting the results of an analysis of the image based on the second feature vector.

2. An information processing device as described in claim 1, comprising an acquisition means for acquiring the images included in a time-series image group in chronological order, wherein the first processing means outputs the current first feature vector obtained by converting language related to the acquired image into a vector, the second processing means outputs the current second feature vector based on the current first feature vector and the second feature vector output before the current one, and the third processing means outputs the results of analysis of the time-series image group based on the current second feature vector.

3. An information processing device according to claim 1 or 2, wherein the second processing means receives explanatory information that explains the operation performed by the third processing means, and outputs the second feature vector based on the explanatory information.

4. The information processing apparatus according to claim 2, wherein said third processing means determines whether or not the current analysis based on said second feature vector is to be the final analysis.

5. An information processing device as described in claim 4, wherein, when it is determined that the analysis based on the current second feature vector is the final analysis, the second processing means outputs a summary of the analysis based on the second feature vector output previously.

6. The information processing device described in claim 2, wherein the third processing means estimates an index indicating to which of a plurality of classes the group of time-series images belongs based on the second feature vector which is based on the first feature vector, the second processing means outputs a second second feature vector which is a vector that converts an explanation regarding the classification of the group of time-series images into one of the plurality of classes based on the index, and the third processing means decides whether to terminate or continue sequential processing which includes at least the acquisition processing by the acquisition means, and if it is decided to terminate the sequential processing, classifies the group of time-series images into one of the plurality of classes.

7. The information processing device according to claim 1, wherein the analysis of the image includes at least one of classifying the image and detecting a predetermined object in the image.

8. The information processing device of claim 2, wherein the analysis of the time series of images includes at least one of classification of the time series of images, detection of a predetermined object in the time series of images, future prediction based on the time series of images, and likelihood ratio estimation based on the time series of images.

9. The information processing device according to claim 6, wherein the index is a likelihood ratio indicating the likelihood that the time-series image group belongs to a certain class of the plurality of classes.

10. A computer-implemented information processing method comprising: outputting a first feature vector obtained by converting language associated with an input image into a vector; outputting a second feature vector obtained by converting a description of the image into a vector based on the first feature vector; and outputting results of an analysis of the image based on the second feature vector.

11. A recording medium having recorded thereon a computer program that causes a computer to execute an information processing method including: outputting a first feature vector obtained by converting language associated with an input image into a vector; outputting a second feature vector obtained by converting a description of the image into a vector based on the first feature vector; and outputting the results of an analysis of the image based on the second feature vector.

Images