Device and system for predicting future event based on artificial intelligence

Abstract

The future event prediction device of the present invention may include a video feature extraction module that extracts a video feature expression from a video; a text feature extraction module that extracts a text feature expression from text related to the video; and a future event prediction module that predicts future events of the video based on the video feature expression and the text feature expression.

Description

Technology Field

[0001] The present invention relates to a future event prediction device, and more specifically, to a device that predicts future events using artificial intelligence. Background Technology

[0002] Recently, with the advancement of Artificial Intelligence (AI) technology, it is being applied to various technological fields. For example, research is being conducted on technologies that use AI to recognize the surrounding environment and predict the behavior of surrounding objects or people. In other words, actions or dialogue that will occur after a certain point in a video can be predicted using AI-based deep learning models; however, existing deep learning models tend to predict these outcomes based only on simple conversations or situations within the video, which leads to a problem of low prediction accuracy. The problem to be solved

[0003] One objective of the present invention is to a future event prediction device capable of increasing the accuracy of predicting future situations and a future event prediction system using the same. means of solving the problem

[0004] A future event prediction device according to one embodiment may include: a video feature extraction module for extracting a video feature expression from a video; a text feature extraction module for extracting a text feature expression from text related to the video; and a future event prediction module for predicting future events of the video based on the video feature expression and the text feature expression.

[0005] Here, the video feature extraction module can extract appearance data and motion data from the video and generate the video feature representation based on the appearance data and the motion data.

[0006] Here, the video feature extraction module may include: a preprocessing unit that extracts appearance data and motion data, respectively, from the video; a first video feature extraction unit that inputs the appearance data into a first machine learning model to extract a first video feature vector; a second video feature extraction unit that inputs the motion data into a second machine learning model to extract a second video feature vector; a first combining unit that combines the first video feature vector and the second video feature vector; and a first encoding unit that performs position encoding on the combined video feature vector to generate the video feature representation.

[0007] Here, the text feature extraction module receives dialogue text and future event text related to the video, respectively, and can generate the text feature representation based on the dialogue text and the future event text.

[0008] Here, the text feature extraction module may include: a text feature extraction unit that inputs the conversation text and the future event text into a third machine learning model to extract a first text feature vector from the conversation text and extract a second text feature vector from the future event text; a second combining unit that combines the first text feature vector and the second text feature vector; and a second encoding unit that performs position encoding on the combined text feature vector to generate the text feature representation.

[0009] Here, the future event prediction module may include a third combining unit that combines the video feature representation and the text feature representation to generate a multimodal feature representation; and a future event prediction unit that inputs the multimodal feature representation into a fourth machine learning model to predict the future event.

[0011] A future event prediction system according to one embodiment may include: an observation bot that transmits user observation information including a video of a user and a recording of a conversation with the user; and a future behavior prediction server that receives the user observation information, extracts video feature expressions and text feature expressions from the user observation information, and predicts the user's future behavior based on the video feature expressions and the text feature expressions.

[0012] Here, the future behavior prediction server includes a future event prediction device, and the future event prediction device can generate the video feature representation by extracting appearance data and motion data from the video, extracting a first video feature vector from the appearance data, extracting a second video feature vector from the motion data, and performing position encoding after combining the first video feature vector and the second video feature vector.

[0013] Here, the future event prediction device can convert the conversation recording into conversation text, generate future event text based on the user observation information, extract a first text feature vector from the conversation text, extract a second text feature vector from the future event text, and generate the text feature representation by combining the first text feature vector and the second text feature vector and then performing position encoding.

[0014] Here, the future event prediction device can generate a multimodal feature representation by combining the video feature representation and the text feature representation, and predict the user's future behavior by inputting the multimodal feature representation into a pre-trained machine learning model.

[0015] Here, the future behavior prediction server can generate user risk information regarding the risk that may befall the user based on the user's predicted future behavior.

[0016] Here, the future behavior prediction server transmits the user risk level information to the observation bot, and the observation bot can notify the user or surrounding people of the user's risk level through an alarm means based on the user risk level information. Effects of the invention

[0017] According to one embodiment of the present invention, a future event prediction device capable of increasing the accuracy of predicting future situations and a future event prediction system using the same may be provided. Brief explanation of the drawing

[0018] FIG. 1 is a block diagram showing a future event prediction device using artificial intelligence according to one embodiment. FIG. 2 is a diagram schematically illustrating the process of generating a multimodal feature representation in a future event prediction device according to one embodiment. FIG. 3 is a diagram showing the configuration of a video feature extraction module (102) according to one embodiment. FIG. 4 is a diagram showing the configuration of a text feature extraction module (104) according to one embodiment. FIG. 5 is a diagram showing the configuration of a future event prediction module (106) according to one embodiment. FIG. 6 is a diagram showing the configuration of a future event prediction system according to one embodiment. FIG. 7 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments. Specific details for implementing the invention

[0019] The embodiments described in this specification are intended to clearly explain the concept of the present invention to those skilled in the art to which the present invention belongs. Therefore, the present invention is not limited to the embodiments described in this specification, and the scope of the present invention should be interpreted to include modifications or variations that do not deviate from the concept of the present invention.

[0020] The terms used in this specification have been selected to be as widely used as possible, taking into account their functions in the present invention; however, these terms may vary depending on the intent of those skilled in the art to which the present invention pertains, case law, or the emergence of new technologies. However, if a specific term is defined and used with an arbitrary meaning, the meaning of that term will be described separately. Accordingly, the terms used in this specification should be interpreted based on their actual meaning and the content throughout this specification, rather than merely their names.

[0021] The drawings attached to this specification are intended to facilitate the explanation of the present invention. Since the shapes depicted in the drawings may be exaggerated as necessary to aid in understanding the present invention, the present invention is not limited by the drawings.

[0022] In this specification, if it is determined that a detailed description of known components or functions related to the present invention may obscure the essence of the present invention, such detailed description will be omitted as necessary.

[0024] FIG. 1 is a block diagram showing a future event prediction device using artificial intelligence according to an embodiment of the present invention. FIG. 2 is a diagram schematically showing the process of generating a multimodal feature representation in a future event prediction device according to an embodiment of the present invention.

[0025] Referring to FIGS. 1 and FIGS. 2, the future event prediction device (100) may include a video feature extraction module (102), a text feature extraction module (104), and a future event prediction module (106).

[0026] The future event prediction device (100) may be a device for predicting future events (events, situations, actions, etc.) based on artificial intelligence. The future event prediction device (100) can predict the next event to occur at a site related to the video (e.g., a site being filmed) when given a video and text related to the video by learning a machine learning model using a video and text related to the video as a data set.

[0027] A video feature extraction module (102) receives a video and can extract video feature representations from the input video. FIG. 3 is a diagram showing the configuration of a video feature extraction module (102) according to an embodiment of the present invention. Referring to FIG. 3, the video feature extraction module (102) may include a preprocessing unit (111), a first video feature extraction unit (113), a second video feature extraction unit (115), a first combining unit (117), and a first encoding unit (119).

[0028] The preprocessing unit (111) receives a video input and can extract appearance data and motion data from the video, respectively. Here, the appearance data may be data related to the background of the video. The motion data may be data related to the movement of objects within the video. In one embodiment, the preprocessing unit (111) can extract appearance data and motion data, respectively, at the 1 FPS (Frame Per Second) level of the input video.

[0029] The first video feature extraction unit (113) receives appearance data and can extract a first video feature vector from the appearance data. To this end, the first video feature extraction unit (113) may include a first machine learning model (113a). The first machine learning model (113a) may be a model that has been pre-trained to extract a first video feature vector from the input appearance data. For example, the first machine learning model (113a) may be a ResNet-152 trained on the ImageNet dataset, but is not limited thereto.

[0030] The second video feature extraction unit (115) receives motion data and can extract a second video feature vector from the motion data. To this end, the second video feature extraction unit (115) may include a second machine learning model (115a). The second machine learning model (115a) may be a model that has been pre-trained to extract a second video feature vector from the input motion data. For example, the second machine learning model (115a) may be a ResNeXt-101 trained on the Kinetics dataset, but is not limited thereto.

[0031] The first combining unit (117) can combine the first video feature vector extracted from the first video feature extraction unit (113) and the second video feature vector extracted from the second video feature extraction unit (115). Specifically, the first combining unit (117) can combine the first video feature vector and the second video feature after calculating the distance between the first video feature vector and the second video feature vector and normalizing it. For example, the first combining unit (117) can combine two video feature vectors by concatenating the normalized first video feature vector and the second video feature vector.

[0032] The first encoding unit (119) can generate a video feature representation for the video by performing positional encoding on the combined video feature vector. That is, when the first video feature vector and the second video feature vector are combined, positional encoding can be performed using positional information regarding the temporal order between video frames to prevent the information regarding the temporal order of the video frames from being lost. In this case, the video feature representation includes both the appearance information and motion information of the video.

[0033] The text feature extraction module (104) receives text related to a video and can extract text feature expressions from the input text. FIG. 4 is a diagram showing the configuration of a text feature extraction module (104) according to an embodiment of the present invention. Referring to FIG. 4, the text feature extraction module (104) may include a text feature extraction unit (121), a second combining unit (123), and a second encoding unit (125).

[0034] The text feature extraction unit (121) can receive conversation text and future event text related to the video, respectively. Here, conversation text may refer to text regarding conversations exchanged between people within the video. Additionally, future event text may refer to text regarding events that are likely to occur in the future within the video (i.e., events that are likely to occur in the future based on the current point in time of the input video frame). Future event text may be input as a pair, consisting of text regarding events that are highly likely to occur in the video (high-probability future event text) and text regarding events that are unlikely to occur in the video (low-probability future event text).

[0035] For example, "Oh Yeah! Maybe a shake" can be entered as dialogue text related to the video. Additionally, regarding future event text related to the video, if there is a woman in the current video wearing a white shirt over black, the high-probability future event text can be "The women in the white short pours the slush into a cup" and the low-probability future event text can be "The women in the white shirt pours the slush into a watermelon rind and passes it to Mark".

[0036] The text feature extraction unit (121) can extract a first text feature vector from dialogue text related to the video and extract a second text feature vector from future event text related to the video. To this end, the text feature extraction unit (121) may include a third machine learning model (121a). The third machine learning model (121a) may be a model pre-trained to receive dialogue text as input to extract a first text feature vector and receive future event text as input to extract a second text feature vector.

[0037] The second combining part (123) can combine the first text feature vector and the second text feature vector. For example, the second combining part (123) can combine two text feature vectors by concatenating the first text feature vector and the second text feature vector.

[0038] The second encoding unit (125) can generate a text feature representation for the video by performing positional encoding on the combined text feature vector. That is, when the first text feature vector and the second text feature vector are combined, positional encoding can be performed using positional information regarding the order of words in the text to prevent information regarding the order or relationship of words in the text from being lost.

[0039] The future event prediction module (106) can predict future events (future events) that will occur in the context of a video based on video feature representations and text feature representations for the video. FIG. 5 is a diagram showing the configuration of a future event prediction module (106) according to an embodiment of the present invention. Referring to FIG. 5, the future event prediction module (106) may include a third coupling part (131) and a future event prediction part (133).

[0040] The third combining unit (131) can receive a video feature expression from the video feature extraction module (102). The third combining unit (131) can receive a text feature expression from the text feature extraction module (104). The third combining unit (131) can generate a multimodal feature expression by combining the video feature expression and the text feature expression. For example, the third combining unit (131) can generate a multimodal feature expression by concatenating the video feature expression and the text feature expression.

[0041] The future event prediction unit (133) receives a multimodal feature representation as input and can predict future events of the video based on the multimodal feature representation. To this end, the future event prediction unit (133) may include a fourth machine learning model (133a). The fourth machine learning model (133a) receives a multimodal feature representation as input and can be trained to predict the event with the highest probability of occurring in the future in the video based on the multimodal feature representation.

[0042] That is, when the fourth machine learning model (133a) receives a multimodal feature representation as input, it can classify future events (future events) that will occur in the video. The future event prediction unit (133) can train the fourth machine learning model (133a) by comparing the future events classified by the fourth machine learning model (133a) with the correct answer value (an event that actually occurred in the future in the video) so that the difference is minimized. At this time, the future event prediction unit (133) can train the fourth machine learning model (133a) using cross-entropy loss.

[0043] The fourth machine learning model (133a) can output classification values ​​for future events based on input multimodal feature representations. For example, the fourth machine learning model (133a) can output probability values ​​for future event A, future event B, future event C, future event D, and future event E, respectively, based on multimodal feature representations. At this time, the fourth machine learning model (133a) can be trained to minimize the difference between the future event with the highest probability value and the correct answer value.

[0044] According to the disclosed embodiment, by receiving a video and text related to the video as input and performing machine learning from a multimodal perspective, it becomes possible to predict events that will occur after the current point in time of the video more accurately. That is, by performing machine learning with the video and text as a dataset, the situation within the video or the user's behavior or intention within the video can be identified more accurately compared to cases where only text or video is provided, and the user's behavior or situation that will occur thereafter can be accurately predicted. In addition, through machine learning, it is possible to infer which event among various future events has a high probability of occurrence.

[0046] FIG. 6 is a diagram showing the configuration of a future event prediction system according to one embodiment of the present invention.

[0047] Referring to FIG. 6, the future event prediction system (200) may include an observation bot (202) and a future behavior prediction server (204). The observation bot (202) is connected to the future behavior prediction server (204) so ​​as to be able to communicate through a communication network (250).

[0048] In one embodiment, the communication network (250) may include the Internet, one or more local area networks, wide area networks, cellular networks, mobile networks, other types of networks, or a combination of these networks.

[0049] The observation bot (202) can perform the role of observing a user (e.g., a person requiring protective observation, such as an elderly person, a patient, or a child) in a certain space. The observation bot (202) can photograph the user by being equipped with a shooting means such as a camera. In addition, the observation bot (202) can record the sound of the space where the user is present by being equipped with a recording means such as a microphone.

[0050] In one embodiment, the observation bot (202) may be configured to enable chatting with the user. That is, the observation bot (202) may be equipped with an AI-based chat function to enable conversation with the user. The observation bot (202) may film the user and record the content of the conversation with the user while conversing with the user. The observation bot (202) may transmit user observation information, including the video of the user filmed and the recording of the user's conversation, to a future behavior prediction server (204).

[0051] The future behavior prediction server (204) can predict the user's future behavior based on user observation information received from the observation bot (202). In one embodiment, the future behavior prediction server (204) may include a future event prediction device (210). The future event prediction device (210) may be the same or similar device as the future event prediction device (100) shown in FIG. 1.

[0052] The future event prediction device (210) can extract video feature representations from user-captured video among user observation information. The future event prediction device (210) can extract a first video feature vector from appearance data among user-captured video, extract a second video feature vector from motion data among user-captured video, and generate a video feature representation by combining the first video feature vector and the second video feature vector and then performing position encoding.

[0053] The future event prediction device (210) can extract text feature expressions from the user's conversation recording among the user observation information. The future event prediction device (210) can convert the user's conversation recording into conversation text. Additionally, the future event prediction device (210) can generate future event text based on the user observation information. In one embodiment, the future event prediction device (210) can analyze the user's age and gender based on the user observation information, and generate future event text, which is text about events likely to occur in the future in the user's video recording, based on the user's age and gender and the user's behavior in the user's video recording.

[0054] The future event prediction device (210) can generate a text feature representation by extracting a first text feature vector from a user's conversation text, extracting a second text feature vector from a future event text, combining the first text feature vector and the second text feature vector, and then performing position encoding.

[0055] The future event prediction device (210) can predict the user's future behavior by combining video feature representations and text feature representations to generate a multimodal feature representation and then inputting it into a pre-trained machine learning model.

[0056] Meanwhile, the future behavior prediction server (204) can calculate the risk level of the user's future behavior when the user's future behavior is predicted by the future event prediction device (210). That is, the future behavior prediction server (204) can calculate the risk level that will befall the user based on the user's predicted future behavior. The future behavior prediction server (204) can transmit information about the risk level that will befall the user (user risk level information) to the observation bot (202). Based on the user risk level information, the observation bot (202) can notify the user or people around them through one or more alarm means among a speaker, a screen, and vibration.

[0057] [Table 1] below is a table showing the risk level and alarm settings according to the user's future behavior according to one embodiment of the present invention.

[0058] Predicted future behavior Risk level Alarm settings staggering 0 ~ 4 safety No need for alarm Falling or slipping 5 ~ 9 minor risk Alarm setting sufficient to recognize dangerous situations collapse 10 ~ 14 serious risk Set the alarm to a volume the user can hear. Falls and slips 15 ~ 20~ Unacceptable risk Set the alarm loud enough for people around you to hear.

[0059] FIG. 7 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in exemplary embodiments. In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0060] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a future event prediction device (100, 210). Additionally, the computing device (12) may be an observation bot (202). Additionally, the computing device (12) may be a future behavior prediction server (204).

[0061] The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) can cause the computing device (12) to operate according to the exemplary embodiment described above. For example, the processor (14) can execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0062] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0063] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0064] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0066] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0067] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0068] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A video feature extraction module that extracts appearance data and motion data from a video and extracts a video feature representation based on the appearance data and the motion data; a text feature extraction module that receives dialogue text and future event text related to the video, respectively, and extracts a text feature representation based on the dialogue text and the future event text; and a future event prediction module that predicts future events of the video based on the video feature representation and the text feature representation, wherein the future event text is: text regarding an event likely to occur in the future within the video, and a text regarding an event likely to occur in the future within the video and a text regarding an event unlikely to occur in the future within the video are input as a pair, and the video feature extraction module comprises: a preprocessing unit that extracts appearance data and motion data from the video, respectively; a first video feature extraction unit that inputs the appearance data into a first machine learning model to extract a first video feature vector; a second video feature extraction unit that inputs the motion data into a second machine learning model to extract a second video feature vector; and a first combining unit that combines the first video feature vector and the second video feature vector. and includes a first encoding unit that generates the video feature representation by performing position encoding on the combined video feature vector, and the text feature extraction module comprises: a text feature extraction unit that inputs the dialogue text and the future event text into a third machine learning model to extract a first text feature vector from the dialogue text and extract a second text feature vector from the future event text; and a second combining unit that combines the first text feature vector and the second text feature vector.A future event prediction device comprising: a second encoding unit that generates a text feature representation by performing position encoding on the combined text feature vector; a future event prediction module comprising: a third combining unit that generates a multimodal feature representation by combining the video feature representation and the text feature representation; and a future event prediction unit that inputs the multimodal feature representation into a fourth machine learning model to predict the future event, wherein the future event prediction unit learns the fourth machine learning model using cross-entropy loss. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 An observation bot that transmits user observation information, including a video of the user and a recording of a conversation with the user; The future behavior prediction server receives the user observation information, extracts a video feature representation and a text feature representation from the user observation information, and predicts the user's future behavior based on the video feature representation and the text feature representation. The future behavior prediction server includes a future event prediction device. The future event prediction device comprises: extracting appearance data and motion data from the video; extracting a first video feature vector from the appearance data and extracting a second video feature vector from the motion data; generating the video feature representation by combining the first video feature vector and the second video feature vector and then performing position encoding; converting the conversation recording into conversation text; generating future event text based on the user observation information; extracting a first text feature vector from the conversation text and extracting a second text feature vector from the future event text; generating the text feature representation by combining the first text feature vector and the second text feature vector and then performing position encoding; generating a multimodal feature representation by combining the video feature representation and the text feature representation; and using cross-entropy loss on the multimodal feature representation A future event prediction system that inputs a learned machine learning model to predict the future behavior of the user, and the future behavior prediction server generates user risk level information regarding the risk level that may befall the user based on the predicted future behavior of the user, transmits the user risk level information to the observation bot, and the observation bot notifies the user or surrounding people of the user's risk level through an alarm means based on the user risk level information. Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 delete

Images