A method and apparatus for predicting behavior using interpretable self-attention
By employing an interpretable self-focusing attention prediction method, the problems of excessive consumption of computing resources and insufficient interpretability in autonomous driving systems are solved. The influencing factors are identified and the prediction accuracy is improved. This method is applicable to autonomous vehicles, aircraft, and robots.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- STRADVISION
- Filing Date
- 2021-04-02
- Publication Date
- 2026-04-14
AI Technical Summary
Existing behavior prediction networks consume excessive computational resources in autonomous driving systems and lack interpretability, failing to clearly identify the features affecting predictions and the areas requiring attention.
An interpretable self-focusing attention prediction method is adopted. Through metadata identification, feature encoding, interpretation module and self-focusing attention module, factors affecting behavior prediction are generated and visualized, thereby improving network performance and the accuracy of specific object behavior prediction.
It achieves enhanced interpretability of behavior prediction networks, reduces computational resource consumption, clarifies influencing factors, and improves prediction accuracy, especially in applications such as autonomous vehicles, aircraft, and robots.
Smart Images
Figure CN115699103B_ABST
Abstract
Description
Technical Field
[0001] This invention claims priority and benefit to U.S. Patent Application No. 63 / 026,424, filed May 18, 2020, and U.S. Patent Application No. 17 / 135,396, filed December 28, 2020, the entire contents of which are incorporated herein by reference.
[0002] This invention relates to behavior prediction, and more specifically, to a method and apparatus for predicting behavior using interpretable self-focusing attention. Background Technology
[0003] Recently, I have been researching methods for object recognition and other tasks using machine learning.
[0004] As one of these machine learning methods, deep learning uses neural networks with multiple hidden layers between the input layer and the output layer, which has high recognition performance.
[0005] Furthermore, deep learning is being applied to various industrial sectors, such as autonomous vehicles, autonomous aircraft, and autonomous robots.
[0006] In particular, in recent years, research has been underway on predicting behavior (actions) by perceiving, localizing, and mapping given video and sensor information, such as predicting the trajectory of an object.
[0007] See Figure 1In existing autonomous vehicles, the behavior prediction device acquires video footage captured during the vehicle's operation and multiple sensor information sensed by the vehicle's sensors via the video and sensor information acquisition module 10. It then acquires multiple images corresponding to each frame of the video and multiple metadata corresponding to the multiple sensor information via the metadata recognition module 20. Each metadata element can include: video image analysis results obtained based on deep learning analysis of multiple video images, such as bounding boxes for multiple objects, traffic lights, and traffic signs; and the vehicle's own posture and map information from the reference sensor information. Furthermore, the behavior prediction device encodes the multiple metadata elements using the feature encoding module 30 to generate multiple features for behavior prediction, and performs learning operations on these features using the deep learning-based behavior prediction module 40 to predict behaviors for multiple objects, i.e., multiple trajectories for multiple objects. Finally, the output module 50 outputs the predicted behaviors for the multiple objects.
[0008] On the other hand, Long Short-Term Memory (LSTM) models, which are typically used to analyze continuous video images, are usually applied to behavior prediction. Recently, Generative Adversarial Networks (GANs) have been used to further improve the behavior prediction performance of LSTM.
[0009] However, deep learning networks are typically unexplainable. That is, it is impossible to know why a certain judgment is made regarding behavior prediction, or which features influence the prediction.
[0010] Therefore, existing behavior prediction networks are improving their performance by providing more complex models and additional techniques, regardless of the features affecting the prediction, thus creating a problem of over-utilization of the computing resources of devices that apply behavior prediction networks.
[0011] Furthermore, with the provision of complex models and additional technologies, devices applying behavioral prediction networks face the challenge of requiring more computational resources. Summary of the Invention
[0012] Technical issues
[0013] The purpose of this invention is to solve all the above-mentioned problems.
[0014] Another objective of this invention is to learn the factors that influence the predictions made by behavior prediction networks.
[0015] Another object of the present invention is to know the areas to be noted when predicting the behavior of a specific object in a behavior prediction network.
[0016] Another object of the present invention is to effectively improve the performance of behavior prediction networks by influencing the factors that affect the predictions made by behavior prediction networks.
[0017] Technical solution
[0018] In order to achieve the above-mentioned objectives of the present invention and to realize the characteristic effects of the present invention described later, the characteristic structure of the present invention is as follows.
[0019] According to an embodiment of the present invention, a method for predicting behavior using interpretable self-focusing attention is provided, comprising: (a) when an image captured by a camera mounted on a device and multiple sensing information sensed by multiple sensors mounted on the device are obtained, a behavior prediction device inputs multiple images corresponding to each frame in the image and multiple sensing information corresponding to each frame to a metadata recognition module, so that the metadata recognition module performs learning operations on the multiple images and multiple sensing information corresponding to each frame to output multiple metadata corresponding to each frame, and inputs the multiple metadata corresponding to each frame to a feature encoding module, so that the feature encoding module encodes each metadata corresponding to each frame and outputs multiple features corresponding to each frame for behavior prediction; (b) the behavior prediction device inputs the multiple images, multiple metadata and multiple features corresponding to each frame to an interpretation module, so that the multiple images, multiple metadata and multiple features corresponding to each frame are learned and processed by the metadata recognition module to output multiple metadata corresponding to each frame; and inputs the multiple metadata corresponding to each frame to a feature encoding module, so that the multiple images, multiple metadata and multiple features corresponding to each frame are learned and processed by the metadata recognition module to output multiple metadata corresponding to each frame for behavior prediction; The explanation module generates multiple explanations corresponding to each frame regarding factors influencing behavior prediction in each frame, inputs multiple images and multiple metadata corresponding to each frame into the self-focusing attention module, and outputs multiple attention maps corresponding to each frame for multiple regions of each object that need to be focused on for behavior prediction in each of the images corresponding to each frame through learning operations. The multiple features and multiple attention maps corresponding to each frame are input into the behavior prediction module, and the behavior prediction module analyzes the multiple features and multiple attention maps corresponding to each frame through learning operations to generate multiple behavior prediction results for the behavior of each object; and (c) the behavior prediction device outputs multiple behaviors of multiple objects corresponding to the behavior prediction results through the output module, and visualizes and outputs the elements influencing behavior prediction by referring to the multiple behavior results and multiple explanations through the visualization module.
[0020] Furthermore, according to an embodiment of the present invention, in the method for predicting behavior, the explanation module and the self-focusing attention module are in a state of being learned by the learning device, which is a state of being learned by the learning device as follows: the learning device inputs multiple learning images corresponding to each learning frame and multiple learning sensor information corresponding to each learning frame to the metadata recognition module, causing the metadata recognition module to output multiple learning metadata corresponding to each learning frame, and inputs the multiple learning metadata corresponding to each learning frame to the feature encoding module, causing the feature encoding module to encode the multiple learning metadata corresponding to each learning frame to output multiple learning features corresponding to each learning frame for behavior prediction, and inputs the multiple learning images and multiple learning sensor information corresponding to each learning frame to the feature encoding module. The learning metadata and multiple learning features are input to the interpretation module, which generates multiple learning interpretations corresponding to each learning frame regarding factors influencing behavior prediction in each learning frame. These multiple learning interpretations and multiple learning metadata are then input to the self-focusing attention module, which analyzes them to output multiple learning attention maps corresponding to multiple learning frames, representing multiple regions that need attention for behavior prediction. The module minimizes multiple interpretation losses referencing the multiple learning interpretations and their corresponding real interpretation data, as well as multiple losses referencing the multiple learning attention maps and their corresponding real attention data.
[0021] In addition, according to an embodiment of the present invention, a method for predicting behavior using interpretable self-focusing attention is provided, wherein in (b), the behavior prediction device enables the interpretation module to generate multiple latent features corresponding to each frame by dimensionality reduction of the image, multiple metadata and multiple features corresponding to each frame by the encoder of the autoencoder, and to reconstruct the multiple latent features corresponding to each frame by the decoder of the autoencoder to generate multiple interpretations corresponding to each frame of multiple factors affecting behavior prediction.
[0022] In addition, according to an embodiment of the present invention, a method for predicting behavior using interpretable self-focusing attention is provided, wherein in (c), the behavior prediction device causes the visualization module to refer to a plurality of the behavior prediction results and a plurality of the interpretations, and to pay attention to at least one target object in each frame of each image that affects the behavior prediction and outputs the result.
[0023] In addition, according to an embodiment of the present invention, a method for predicting behavior using interpretable self-focusing attention is provided, wherein in (b), the behavior prediction device enables the interpretation module to perform learning operations on multiple images, multiple metadata and multiple features corresponding to each frame through an autoencoder to generate multiple semantic segmentation images corresponding to each frame, and then generates an interpretation image corresponding to each frame that displays multiple elements that affect the behavior prediction through instance-level attention.
[0024] In addition, according to an embodiment of the present invention, a method for predicting behavior using interpretable self-focusing attention is provided, wherein in (b), the behavior prediction device causes the interpretation module to perform learning operations on a plurality of metadata corresponding to each frame, and generates a plurality of decision trees based on all the metadata associated with objects in the plurality of images corresponding to each frame.
[0025] In addition, according to an embodiment of the present invention, a method for predicting behavior using interpretable self-focusing attention is provided, wherein in (c), the behavior prediction device causes the visualization module to refer to a plurality of decision trees and a plurality of interpretations to output state information about at least one target object in each frame that affects the behavior prediction.
[0026] In addition, according to an embodiment of the present invention, a method for predicting behavior using interpretable self-focusing attention is provided, wherein in (a), the behavior prediction device inputs a plurality of images and a plurality of sensor information corresponding to each frame into the metadata recognition module, so that the metadata recognition module detects the surrounding environmental information of the plurality of devices through the perception module and detects the positioning information of the devices through the positioning and mapping module.
[0027] In addition, according to an embodiment of the present invention, a method for predicting behavior using interpretable self-focusing attention is provided, wherein the plurality of metadata corresponding to each frame includes at least a plurality of object bounding boxes corresponding to a plurality of objects, a plurality of pose information of the device, and map information corresponding to the area where the device is located.
[0028] In addition, according to one embodiment of the present invention, a method for predicting behavior using interpretable self-focusing attention is provided, wherein the behavior prediction module includes a recurrent neural network (RNN), which is composed of at least one of a long short-term memory (LSTM) algorithm and a long short-term memory-generative adversarial network (GAN) algorithm.
[0029] Additionally, according to an embodiment of the present invention, a behavior prediction device using interpretable self-focusing attention is provided, comprising: at least one memory storing a plurality of instructions for predicting behavior using interpretable self-focusing attention; and a processor that uses interpretable self-focusing attention to predict behavior according to the instructions, the processor performing the following processing: (I) when a video captured by a camera mounted on a device and a plurality of sensing information sensed by a plurality of sensors mounted on the device are obtained, the behavior prediction device inputs a plurality of images corresponding to each frame in the video and a plurality of the sensing information corresponding to each frame to a metadata recognition module, so that the metadata recognition module performs learning operations on the plurality of images and the plurality of sensing information corresponding to each frame to output a plurality of metadata corresponding to each frame, and inputs the plurality of metadata corresponding to each frame to a feature encoding module, so that the feature encoding module encodes each of the metadata corresponding to each frame and outputs a plurality of features corresponding to each frame for behavior prediction; (II) the behavior prediction device inputs a plurality of images and the plurality of sensing information sensed by a camera mounted on a device and a plurality of the sensing information sensed by a plurality of sensors mounted on the device to a metadata recognition module, the processor performing the following processing: (I) when a video captured by a camera mounted on a device and a plurality of sensing information sensed by a plurality of sensors mounted on the device are obtained, the behavior prediction device inputs a plurality of images corresponding to each frame and a plurality of the sensing information sensed by a plurality of sensors mounted on the device to a metadata recognition module, the metadata recognition module performs learning operations on the plurality of images and the plurality of sensing information sensed by a plurality of sensors mounted on the device to output a plurality of metadata corresponding to each frame, and inputs a plurality of the metadata corresponding to each frame to a feature encoding module, the feature encoding module encodes each of the metadata corresponding to each frame and outputs a plurality of features corresponding to each frame for behavior prediction; (II) the behavior prediction device inputs a plurality of images and the plurality of sensing information sensed by a camera mounted on the device to a metadata recognition module, the processor performing learning operations on the plurality of images and the plurality of sensing information sensed by a plurality of sensors mounted on the device to output The multiple images, multiple metadata, and multiple features corresponding to each frame are input to the interpretation module, which generates multiple interpretations for each frame regarding factors influencing behavior prediction. The multiple images and multiple metadata corresponding to each frame are input to the self-focusing attention module, which outputs multiple attention maps corresponding to each frame for multiple regions of each object that need to be focused on for behavior prediction in each of the images corresponding to each frame through learning operations. The multiple features and multiple attention maps corresponding to each frame are input to the behavior prediction module, which analyzes the multiple features and multiple attention maps corresponding to each frame through learning operations to generate multiple behavior prediction results for the behavior of each object. (III) The behavior prediction device outputs multiple behaviors of multiple objects corresponding to the behavior prediction results through the output module, and visualizes and outputs the elements influencing behavior prediction by referring to the multiple behavior results and multiple interpretations through the visualization module.
[0030] Furthermore, according to an embodiment of the present invention, a behavior prediction device using interpretable self-focusing attention is provided. In the process, the interpretation module and the self-focusing attention module are in a state of being learned by a learning device, which is a state of being learned by the learning device as follows: the learning device inputs multiple learning images corresponding to each learning frame and multiple learning sensor information corresponding to each learning frame to the metadata recognition module, causing the metadata recognition module to output multiple learning metadata corresponding to each learning frame, and inputs the multiple learning metadata corresponding to each learning frame to the feature encoding module, causing the feature encoding module to encode the multiple learning metadata corresponding to each learning frame to output multiple learning features corresponding to each learning frame for behavior prediction, and inputs the multiple learning metadata corresponding to each learning frame to the feature encoding module, causing the feature encoding module to encode the multiple learning metadata corresponding to each learning frame to output multiple learning features corresponding to each learning frame for behavior prediction, and inputs the multiple learning metadata corresponding to each learning frame to the feature encoding module. The learning image, multiple learning metadata, and multiple learning features are input to the interpretation module, causing the interpretation module to generate multiple learning interpretations corresponding to each learning frame regarding factors influencing behavior prediction in each learning frame. The multiple learning interpretations and multiple learning metadata corresponding to each learning frame are input to the self-focusing attention module, causing the self-focusing attention module to analyze the multiple learning interpretations and multiple learning metadata corresponding to each learning frame to output multiple learning attention maps corresponding to multiple learning frames for multiple regions that need to be focused on for behavior prediction. The self-focusing attention module minimizes multiple interpretation losses referencing multiple learning interpretations and their corresponding multiple interpretation real data, as well as multiple losses referencing multiple learning attention maps and their corresponding multiple attention real data.
[0031] In addition, according to an embodiment of the present invention, a behavior prediction device using interpretable self-focusing attention is provided. In the (II) process, the behavior prediction device enables the interpretation module to generate multiple latent features corresponding to each frame by dimensionality reduction of the image, multiple metadata and multiple features corresponding to each frame by the encoder of the autoencoder, and to reconstruct the multiple latent features corresponding to each frame by the decoder of the autoencoder to generate multiple interpretations corresponding to each frame of multiple factors affecting behavior prediction.
[0032] In addition, according to an embodiment of the present invention, a behavior prediction device using interpretable self-focusing attention is provided, wherein in the (III) process, the behavior prediction device causes the visualization module to refer to a plurality of the behavior prediction results and a plurality of the interpretations, and to pay attention to at least one target object in each frame of each image that affects the behavior prediction and outputs the result.
[0033] In addition, according to an embodiment of the present invention, a behavior prediction device using interpretable self-focusing attention is provided. In the (II) process, the behavior prediction device enables the interpretation module to perform learning operations on multiple images, multiple metadata and multiple features corresponding to each frame through the autoencoder to generate multiple semantic segmentation images corresponding to each frame, and then generates an interpretation image corresponding to each frame that displays multiple elements affecting the behavior prediction through instance-level attention.
[0034] In addition, according to an embodiment of the present invention, a behavior prediction device using interpretable self-focusing attention is provided, wherein in the (II) process, the behavior prediction device causes the interpretation module to perform learning operations on a plurality of metadata corresponding to each frame, and generates a plurality of decision trees based on all the metadata associated with objects in the plurality of images corresponding to each frame.
[0035] In addition, according to an embodiment of the present invention, a behavior prediction device using interpretable self-focusing attention is provided, wherein in the (III) process, the behavior prediction device causes the visualization module to refer to a plurality of decision trees and a plurality of interpretations to output state information about at least one target object in each frame that affects the behavior prediction.
[0036] In addition, according to an embodiment of the present invention, a behavior prediction device using interpretable self-focusing attention is provided. In the (I) process, the behavior prediction device inputs a plurality of images and a plurality of sensor information corresponding to each frame into the metadata recognition module, so that the metadata recognition module detects the surrounding environmental information of the plurality of devices through the perception module and detects the positioning information of the devices through the positioning and mapping module.
[0037] In addition, according to an embodiment of the present invention, a behavior prediction device using interpretable self-focusing attention is provided, wherein the plurality of metadata corresponding to each frame includes at least a plurality of object bounding boxes corresponding to a plurality of objects, a plurality of pose information of the device, and map information corresponding to the area where the device is located.
[0038] In addition, according to an embodiment of the present invention, a behavior prediction device using interpretable self-focusing attention is provided, wherein the behavior prediction module includes a recurrent neural network (RNN), which is composed of at least one of the Long Short-Term Memory (LSTM) algorithm and the Long Short-Term Memory-Generative Adversarial Network (GAN) algorithm.
[0039] In addition, the present invention also provides a computer-readable recording medium for recording a computer program for performing the method of the present invention.
[0040] Beneficial effects
[0041] This invention can reveal the factors that influence the predictions made by behavior prediction networks.
[0042] This invention can identify areas of attention when predicting the behavior of a specific object in a behavior prediction network.
[0043] This invention can effectively improve the performance of behavior prediction networks by considering the factors that influence their predictions. Attached Figure Description
[0044] The following drawings, which are used to describe embodiments of the present invention, are only a part of the embodiments of the present invention, and those skilled in the art to which the present invention pertains (hereinafter referred to as "skilled persons") can obtain other drawings based on these drawings without any creative work.
[0045] Figure 1 This is a schematic block diagram of an existing behavior prediction device.
[0046] Figure 2 This is a schematic diagram of a behavior prediction device that uses interpretable self-focusing attention to predict behavior according to an embodiment of the present invention.
[0047] Figure 3 This is a schematic block diagram of a behavior prediction device that uses interpretable self-focusing attention to predict behavior according to an embodiment of the present invention.
[0048] Figure 4 This is an example of generating an explanation for behavior prediction in a method for using interpretable self-focusing attention to predict behavior according to an embodiment of the present invention.
[0049] Figure 5 This is another example of generating an explanation for behavior prediction in a method for using interpretable self-focusing attention to predict behavior according to an embodiment of the present invention.
[0050] Figure 6a and Figure 6b A process diagram for generating self-focused attention for behavior prediction in a method for predicting behavior using interpretable self-focused attention according to an embodiment of the present invention.
[0051] Figure 7 This is a state diagram for visualizing the interpretation used in behavior prediction in a method for predicting behavior using interpretable self-focusing attention, according to an embodiment of the present invention.
[0052] Figure 8This is a state diagram of the learning interpretation module and the self-focusing attention module in a behavior prediction device using interpretable self-focusing attention according to an embodiment of the present invention. Detailed Implementation
[0053] The following detailed description of the invention is given with reference to the accompanying drawings, which illustrate specific embodiments in which the invention can be practiced to explain the objectives, technical solutions, and advantages of the invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention.
[0054] Furthermore, in the description and claims of this invention, the term "comprising" and its variations are not intended to exclude other technical features, additions, components, or steps. Other objects, advantages, and features of this invention will be apparent to those skilled in the art, in part from this specification and in part from practice of the invention. The following illustrations and figures are provided as examples and are not intended to limit the scope of the invention.
[0055] Furthermore, the present invention includes all possible combinations of the embodiments shown in this specification. It should be understood that the various embodiments of the present invention, though different, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in one embodiment by other embodiments without departing from the spirit and scope of the invention. Furthermore, it should be understood that the position or configuration of the components in each disclosed embodiment may be varied without departing from the spirit and scope of the invention. Therefore, the detailed description that follows is not intended to limit the invention; the scope of the invention should be defined by all scopes equivalent to those of the claims and the appended claims, provided that appropriate description is possible. Similar reference numerals in the drawings indicate the same or similar functions in several respects.
[0056] To enable those skilled in the art to readily implement the present invention, preferred embodiments of the invention will now be described in detail with reference to the accompanying drawings.
[0057] Figure 2 This is a schematic diagram of a behavior prediction device using explainable self-focused attention according to an embodiment of the present invention. See also: Figure 2 The behavior prediction device 1000 may include: a memory 1001 storing a plurality of instructions for predicting behavior using interpretable self-focusing attention; and a processor 1002 that uses interpretable self-focusing attention to predict behavior according to the plurality of instructions.
[0058] Specifically, the behavior prediction device 1000 can typically achieve the desired system performance using a combination of computing devices (e.g., computer processors, memory, storage devices, input and output devices, and other components that may include conventional computing devices; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that enable the computing devices to operate in a particular manner).
[0059] Additionally, the processor of a computing device may include hardware configurations such as a microprocessor unit (MPU) or central processing unit (CPU), cache memory, and a data bus. Furthermore, the computing device may include a software configuration such as an operating system and applications for executing specific purposes.
[0060] However, it is not excluded that the computing device may include an integrated processor in the form of an integrated medium, processor, and memory for the purpose of implementing the present invention.
[0061] Additionally, see Figure 3 The behavior prediction device may include an image and sensor information acquisition module 100, a metadata recognition module 200, a feature encoding module 300, a behavior prediction module 400, an output module 500, an interpretation module 600, a self-focusing attention module 700, and a visualization module 800.
[0062] The process by which the behavior prediction device 1000 configured as described above performs behavior prediction using interpretable self-focusing attention will be described below. In the following description, the device for predicting behavior in response to surrounding objects will be primarily directed at autonomous vehicles, but the invention is not limited thereto; it is equally applicable to autonomous aircraft, autonomous robots, and mobile devices, etc.
[0063] See Figure 3 The behavior prediction device 1000 can acquire video captured by a camera installed on the device and multiple sensing information sensed by multiple sensors installed on the device.
[0064] At this time, the behavior prediction device 1000 can acquire video captured by the camera and multiple sensor information sensed by multiple sensors of the autonomous vehicle while it is driving through the image and sensor information acquisition module 100.
[0065] On the other hand, although the above description describes acquiring images via a camera, the present invention is not limited thereto; images can also be acquired using lidar, radar, or sensor fusion technology. Furthermore, the acquired video may include environmental information of the area corresponding to a specific viewing angle of the device, or it may include environmental information of the entire surrounding environment of the device.
[0066] Next, when the video and multiple sensor information are obtained, the behavior prediction device 1000 can input multiple images corresponding to each frame in the video and multiple sensor information corresponding to each frame into the metadata recognition module 200, so that the metadata recognition module 200 can perform learning operations on the multiple images and multiple sensor information corresponding to each frame to output multiple metadata corresponding to each frame.
[0067] Specifically, the behavior prediction device 1000 can input multiple images and multiple sensor information corresponding to each frame into the metadata recognition module 200, so that the metadata recognition module 200 can detect the surrounding environmental information of multiple devices through the perception module and detect the location information of the devices through the localization and mapping module.
[0068] Furthermore, the perception module may include a deep learning-based object detection network, a deep learning-based segmentation network, etc., and the metadata recognition module 200 can generate multiple metadata corresponding to each frame by using deep learning-based video image analysis results and multiple sensor information.
[0069] In addition, the metadata corresponding to each frame may include at least one object bounding box corresponding to the multiple objects, multiple pose information of the device, traffic lights, traffic signs, and map information corresponding to the area where the device is located, but is not limited to these, and may include various information that can be used for behavior prediction.
[0070] Next, the behavior prediction device 1000 can input multiple metadata corresponding to each frame into the feature encoding module 300, so that the feature encoding module 300 encodes each metadata corresponding to each frame and outputs multiple features corresponding to each frame for behavior prediction.
[0071] Next, the behavior prediction device 1000 can input multiple images, multiple metadata and multiple features corresponding to each frame into the interpretation module 600, so that the interpretation module 600 generates multiple interpretations corresponding to each frame about the factors affecting behavior prediction on each frame.
[0072] Specifically, see Figure 4 The behavior prediction device 100 can enable the interpretation module 600 to generate multiple latent features corresponding to each frame by dimensionality reduction of the image, multiple metadata and multiple features corresponding to each frame through the encoder of the autoencoder, and to reconstruct the multiple latent features corresponding to each frame through the decoder of the autoencoder to generate multiple interpretations corresponding to each frame of multiple factors affecting behavior prediction.
[0073] That is, the behavior prediction device 1000 can enable the interpretation module 600 to perform learning operations on multiple images, multiple metadata and multiple features corresponding to each frame through an autoencoder to generate multiple semantic segmentation images corresponding to each frame, and then generate interpretation images corresponding to each frame by displaying multiple factors affecting the behavior prediction through instance-wise attention.
[0074] Additionally, see Figure 5 The behavior prediction device 1000 enables the interpretation module 600 to perform learning operations on the multiple metadata corresponding to each frame, and generate multiple decision trees based on all the multiple metadata associated with objects in the multiple images corresponding to each frame.
[0075] As an example, when an object is identified as a "cat", the explanation module 600 generates a decision tree from multiple factors that led to the object being identified as a "cat", such as color, shape, etc. By referring to the decision tree, the user can not only understand the process by which the learning network identifies the object as a "cat", but also easily identify the reasons why the learning network makes mistakes in the recognition process.
[0076] Next, see Figure 3 The behavior prediction device 1000 inputs multiple images and multiple metadata corresponding to each frame to the self-focusing attention module 700, so that the self-focusing attention module 700 outputs multiple attention maps corresponding to each frame for multiple regions of each object that need to be focused on in each of the images corresponding to each frame for behavior prediction through learning operations.
[0077] As an example, see Figure 6a and Figure 6b Based on deep learning, the system analyzes multiple video images and metadata corresponding to each frame captured by cameras during the operation of autonomous vehicles. Figure 6a As shown, when a bus is waiting due to pedestrian movement, it can pay attention to the pedestrians, and as... Figure 6b As shown, when a bus starts moving and pedestrians are waiting, they can pay attention to the bus.
[0078] Next, the behavior prediction device 1000 inputs multiple features and multiple attention maps corresponding to each frame into the behavior prediction module 400, so that the behavior prediction module 400 analyzes the multiple features and multiple attention maps corresponding to each frame through learning operations to generate multiple behavior prediction results predicting the behavior of each object.
[0079] On the other hand, unlike predicting multiple trajectories for all objects in multiple video images, the behavior prediction device 1000 can also predict multiple trajectories for only the multiple objects of interest in the attention map.
[0080] At this time, the behavior prediction module 400 may include a recurrent neural network (RNN), which is composed of at least one of the Long Short-Term Memory (LSTM) algorithm and the Long Short-Term Memory-Generative Adversarial Network (GAN) algorithm.
[0081] Next, the behavior prediction device 1000 outputs multiple behaviors of the multiple objects corresponding to the behavior prediction results through the output module 500. Simultaneously, the visualization module 800 visualizes and outputs the factors influencing the behavior prediction, referencing the multiple behavior results and multiple explanations.
[0082] Specifically, the behavior prediction device 100 causes the visualization module 800 to refer to multiple behavior prediction results and multiple interpretations, and to notice and output at least one target object in each frame of each image that affects the behavior prediction.
[0083] In addition, the behavior prediction device 1000 causes the visualization module 800 to refer to multiple decision trees and multiple interpretations to output state information about at least one target object in each frame that affects behavior prediction.
[0084] As an example, see Figure 7The visualization module 800 can visualize the status of "pedestrian" and "target vehicle" based on multiple interpretations of multiple decision trees, or visualize and output attention graphs about "pedestrian" and "target vehicle".
[0085] On the other hand, the interpretation module 600 and the self-focusing attention module 700 of the behavior prediction device 1000 can be in a state of being learned by the learning device.
[0086] That is, see Figure 8 The learning device 200 inputs multiple learning images and multiple learning sensor information corresponding to each learning frame to the metadata recognition module 200, causing the metadata recognition module 200 to output multiple learning metadata corresponding to each learning frame. The learning metadata corresponding to each learning frame is then input to the feature encoding module 300, causing the feature encoding module 300 to encode the multiple learning metadata corresponding to each learning frame to output multiple learning features for behavior prediction corresponding to each learning frame. The learning device 200 also inputs multiple learning images, multiple learning metadata, and multiple learning sensor information corresponding to each learning frame. The learning features are input to the interpretation module 600, which generates multiple learning interpretations corresponding to each learning frame regarding factors influencing behavior prediction in each learning frame. The multiple learning interpretations and multiple learning metadata corresponding to each learning frame are input to the self-focusing attention module, which analyzes these interpretations and metadata to output multiple learning attention maps corresponding to multiple learning frames, representing multiple regions that need attention for behavior prediction. The module minimizes multiple interpretation losses (referring to multiple learning interpretations and their corresponding ground truth) and multiple losses (referring to multiple learning attention maps and their corresponding ground truth).
[0087] At this time, the learning device 2000 may include: a memory that stores a plurality of instructions for training the interpretation module 600 and the self-focusing attention module 700 of the behavior prediction device 1000; and a processor that executes operations for training the interpretation module 600 and the self-focusing attention module 700 of the behavior prediction device 1000 according to the plurality of instructions stored in the memory.
[0088] Specifically, the learning device 2000 can typically achieve the required system performance using a combination of computing devices (e.g., computer processors, memory, storage devices, input and output devices, and other components that may include conventional computing devices; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that enable the computing device to operate in a particular manner).
[0089] Additionally, the processor of a computing device may include hardware configurations such as a microprocessor unit (MPU) or central processing unit (CPU), cache memory, and a data bus. Furthermore, the computing device may include a software configuration such as an operating system and applications for executing specific purposes.
[0090] However, it is not excluded that the computing device may include an integrated processor in the form of an integrated medium, processor, and memory for the purpose of implementing the present invention.
[0091] The following will refer to Figure 8 The method of training the interpretation module 600 and the self-focusing attention module 700 of the behavior prediction device 1000 using the learning device 2000 configured as described above will be described in more detail below. (The following will omit information from...) Figure 3 The description is a detailed description of the parts that are easy to understand.
[0092] First, the learning device 2000 can obtain multiple learning images corresponding to each learning frame and multiple learning sensor information corresponding to each learning frame, which are obtained by the image and sensor information acquisition module 100.
[0093] Furthermore, the learning device 2000 can input multiple learning images corresponding to each learning frame and multiple learning sensor information corresponding to each learning frame into the metadata recognition module 200, so that the metadata recognition module 200 outputs multiple learning metadata corresponding to each learning frame.
[0094] Subsequently, the learning device 2000 can input multiple learning metadata corresponding to each learning frame into the feature encoding module 300, so that the feature encoding module 300 encodes the multiple metadata corresponding to each learning frame and outputs multiple learning features corresponding to each learning frame for behavior prediction.
[0095] Next, the learning device 2000 can input multiple learning images, multiple learning metadata, and multiple learning features corresponding to each learning frame into the interpretation module 600, so that the interpretation module 600 generates multiple learning interpretations corresponding to each learning frame regarding the factors that affect the behavior prediction in each learning frame.
[0096] Next, the learning device 2000 can input multiple learning explanations and multiple learning metadata corresponding to each learning frame into the self-focusing attention module 700, so that the self-focusing attention module 700 analyzes the multiple learning explanations and multiple learning metadata corresponding to each learning frame to output multiple learning attention maps corresponding to the multiple learning frames for multiple areas that need to be focused on for behavior prediction.
[0097] Subsequently, the learning device 2000 can train the explanation module 600 and the self-focusing attention module 700 to minimize multiple explanation losses referencing multiple learning explanations and their corresponding multiple real explanation data, and multiple losses referencing multiple learning attention maps and their corresponding multiple real attention data, respectively. At this time, the learning device 2000 can cause the first loss layer 910 to generate multiple explanation losses referencing multiple learning explanations and their corresponding multiple real explanation data, and can cause the second loss layer 920 to generate multiple attention losses referencing multiple learning attention maps and their corresponding multiple real attention data.
[0098] On the other hand, even though the interpretation module 600 and the self-focusing attention module 700 are trained by using multiple learning images and their corresponding multiple learning sensor information as described above, it is also possible to generate a learning data by using multiple learning images and their corresponding multiple learning sensor information, the learning metadata and the learning features, and to train the interpretation module 600 and the self-focusing attention module 700 using the generated learning data.
[0099] Furthermore, the embodiments described above according to the present invention can be implemented and recorded in a computer-readable recording medium in the form of program instructions executable by various computer components. The computer-readable recording medium may include individual or combined program instructions, data files, data structures, etc. The program instructions recorded in the computer-readable recording medium may be specifically designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; and hardware devices specifically configured for storing and executing program instructions, such as ROMs, RAMs, flash memory, etc. Examples of program instructions include not only machine language code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as at least one software module to perform the processing according to the present invention, and vice versa.
[0100] In the foregoing, the invention has been described with reference to specific matters such as particular components, as well as limited embodiments and accompanying drawings. However, this is only to help to understand the invention more fully, and the invention is not limited to the embodiments described above. Various modifications and variations can be designed by those skilled in the art based on these descriptions.
[0101] Therefore, the spirit of the present invention should not be limited to the above embodiments. Except for the appended claims, all modifications that are equivalent or analogous to these claims should be included within the spirit and scope of the present invention.
Claims
1. A method for predicting trajectories, characterized in that, The method is configured to predict trajectories using interpretable self-focusing attention, and includes the following steps: Test videos are captured by a camera installed on the device; Obtain multiple test sensing information sensed by multiple test sensors installed on the device; (a) The trajectory prediction device performs the following process: (a) (i) Input multiple test images corresponding to each test frame in the test video and multiple test sensor information corresponding to each test frame into the metadata recognition module, so that the metadata recognition module performs learning operations on the multiple test images and multiple test sensor information corresponding to each test frame to output multiple test metadata corresponding to each test frame. (a) (ii) Input the plurality of test metadata corresponding to each test frame into the feature encoding module, so that the feature encoding module encodes each of the test metadata corresponding to each test frame to output the plurality of test features corresponding to each test frame; (b) The trajectory prediction device performs the following process: (b)(i) Input the plurality of test images, the plurality of test metadata and the plurality of test features corresponding to each test frame into the interpretation module, so that the interpretation module generates a plurality of test interpretations corresponding to each test frame regarding the test factors that affect the prediction of the test trajectory on each test frame. (b) (ii) Input the plurality of test images and the plurality of test metadata corresponding to each test frame into the self-focusing attention module, so that the self-focusing attention module outputs, through learning operations, a plurality of test attention maps corresponding to each test frame for a plurality of test regions of each test object that needs to be focused on for test trajectory prediction in each of the test images corresponding to each test frame. (b) (iii) Input the multiple test features and multiple test attention maps corresponding to each test frame into the trajectory prediction module, so that the trajectory prediction module analyzes the multiple test features and multiple test attention maps corresponding to each test frame through learning operations, predicts the trajectory of each test object, and generates multiple test trajectory predictions. (c) The trajectory prediction device performs the following process: (c) (i) Output the test trajectory results corresponding to the multiple test trajectory predictions for the multiple test objects through the output module. (c) (ii) By referring to multiple test trajectory predictions and multiple test interpretations, at least one target object affecting trajectory prediction is marked as the test region in each test image, and each marked test image is output, thereby allowing the visualization module to visualize and output the test factors affecting trajectory prediction.
2. The method according to claim 1, wherein: The explanation module and the self-focusing attention module are in the following states that have been learned by the learning device: The learning device: (i) Input multiple learning images corresponding to each learning frame and multiple learning sensor information corresponding to each learning frame into the metadata recognition module, so that the metadata recognition module outputs multiple learning metadata corresponding to each learning frame. (ii) Inputting the plurality of learning metadata corresponding to each learning frame into the feature encoding module, so that the feature encoding module encodes the plurality of learning metadata corresponding to each learning frame to output the plurality of learning features corresponding to each learning frame for trajectory prediction. (iii) Inputting multiple learning images, multiple learning metadata, and multiple learning features corresponding to each learning frame into the interpretation module, so that the interpretation module generates multiple learning interpretations corresponding to each learning frame regarding the factors affecting trajectory prediction on each learning frame. (iv) Input the multiple learning explanations and multiple learning metadata corresponding to each learning frame into the self-focusing attention module, so that the self-focusing attention module analyzes the multiple learning explanations and multiple learning metadata corresponding to each learning frame to output multiple learning attention maps corresponding to the multiple learning frames for multiple regions that need to be focused on for trajectory prediction, and (v) (v-1) Minimize multiple interpretation losses that refer to multiple learning interpretations and their corresponding multiple interpretation real data, and (v-2) Minimize multiple losses that refer to multiple learning attention maps and their corresponding multiple attention real data.
3. The method according to claim 1, wherein: In (b) described above, The trajectory prediction device enables the interpretation module to: (i) Generating multiple latent test features corresponding to each test frame by dimensionality reduction of the test image, multiple test metadata, and multiple test features corresponding to each test frame using an autoencoder, and (ii) Reconstruct multiple test latent features corresponding to each test frame using the decoder of the autoencoder to generate multiple test interpretations corresponding to each test frame for multiple test factors that affect the prediction of the test trajectory.
4. The method according to claim 1, wherein: In (b) described above, The trajectory prediction device causes the interpretation module (i) (i-1) to perform learning operations on multiple test images, multiple test metadata and multiple test features corresponding to each test frame to generate multiple test semantic segmentation images corresponding to each test frame, and (i-2) to identify instance-level regions of interest in the test semantic segmentation images by an autoencoder, and then (ii) to generate a test interpretation image corresponding to each test frame, and to mark multiple test factors that affect the test trajectory prediction with reference to the results of (i-2) above.
5. The method according to claim 1, wherein: In (b) described above, The trajectory prediction device enables the interpretation module to perform learning operations on multiple test metadata corresponding to each test frame, and generate multiple test decision trees based on all the multiple test metadata associated with the test object in the multiple test images corresponding to each test frame.
6. The method according to claim 5, wherein: In (c), The trajectory prediction device causes the visualization module to reference multiple test decision trees and multiple test interpretations to output state information about at least one target object in each test frame that affects the test trajectory prediction.
7. The method according to claim 1, wherein: In (a), The trajectory prediction device inputs multiple test images and multiple test sensing information corresponding to each frame into the metadata recognition module, so that the metadata recognition module (1) detects the environmental information around the multiple devices through the perception module, and (2) detects the positioning information of the devices through the positioning and mapping module.
8. The method according to claim 1, wherein: The test metadata corresponding to each frame includes at least multiple object bounding boxes corresponding to multiple test objects, multiple pose information of the device, and map information corresponding to the area where the device is located.
9. The method according to claim 1, wherein: The trajectory prediction module includes a recurrent neural network, which consists of at least one of a long short-term memory algorithm and a long short-term memory-generative adversarial network algorithm.
10. A trajectory prediction device for predicting trajectories, characterized in that, The trajectory prediction device is configured to predict trajectories using interpretable self-focusing attention and is mounted on the device. The trajectory prediction device includes: A camera, which is mounted on the device; At least one sensor is mounted on the device; At least one memory storing a plurality of instructions for predicting trajectories using the trajectory prediction device; and A processor that uses the trajectory prediction device to predict a trajectory according to the instructions. The processor performs the following steps: Step (I): (I)(i) Inputting multiple test images corresponding to each test frame in the test video captured by the camera and multiple test sensing information corresponding to each test frame sensed by the at least one sensor when the device moves into the metadata recognition module, so that the metadata recognition module performs learning operations on the multiple test images and multiple test sensing information corresponding to each test frame to output multiple test metadata corresponding to each test frame. (I) (ii) Input the plurality of test metadata corresponding to each test frame into the feature encoding module, so that the feature encoding module encodes each of the test metadata corresponding to each test frame and outputs the plurality of test features corresponding to each test frame; Step (II): (II) (i) Input the plurality of test images, the plurality of test metadata and the plurality of test features corresponding to each test frame into the interpretation module, so that the interpretation module generates a plurality of test interpretations corresponding to each test frame regarding the test factors that affect the prediction of the test trajectory on each test frame. (II) (ii) Input the multiple test images and multiple test metadata corresponding to each test frame into the self-focusing attention module, so that the self-focusing attention module outputs multiple test attention maps corresponding to each test frame for multiple test regions of each test object that needs to be focused on for test trajectory prediction in each test image corresponding to each test frame through learning operations. (II) (iii) Input the multiple test features and multiple test attention maps corresponding to each test frame into the trajectory prediction module, so that the trajectory prediction module analyzes the multiple test features and multiple test attention maps corresponding to each test frame through learning operations, predicts the trajectory of each test object, and generates multiple test trajectory predictions. Step (III): (III) (i) Output the test trajectory results corresponding to the multiple test trajectory predictions of the multiple test objects through the output module. (III) (ii) By referring to multiple test trajectory predictions and multiple test interpretations, at least one target object affecting trajectory prediction is marked as the test region in each test image, and each marked test image is output, thereby allowing the visualization module to visualize and output the test factors affecting trajectory prediction.
11. The trajectory prediction device according to claim 10, wherein: In the process, The explanation module and the self-focusing attention module are in the following states that have been learned by the learning device: the learning device (i) Input multiple learning images corresponding to each learning frame and multiple learning sensor information corresponding to each learning frame into the metadata recognition module, so that the metadata recognition module outputs multiple learning metadata corresponding to each learning frame. (ii) Inputting the plurality of learning metadata corresponding to each learning frame into the feature encoding module, so that the feature encoding module encodes the plurality of learning metadata corresponding to each learning frame to output the plurality of learning features corresponding to each learning frame for trajectory prediction. (iii) Inputting multiple learning images, multiple learning metadata, and multiple learning features corresponding to each learning frame into the interpretation module, so that the interpretation module generates multiple learning interpretations corresponding to each learning frame regarding the factors affecting trajectory prediction in each learning frame. (iv) Input the plurality of learning interpretations and plurality of learning metadata corresponding to each of the learning frames into the self-focusing attention module, so that the self-focusing attention module analyzes the plurality of learning interpretations and plurality of learning metadata corresponding to each of the learning frames to output a plurality of learning attention maps corresponding to the plurality of learning frames for the plurality of regions that need to be focused on for trajectory prediction, and (v) (v-1) Minimize multiple interpretation losses that refer to multiple learning interpretations and their corresponding multiple interpretation real data, and (v-2) Minimize multiple losses that refer to multiple learning attention maps and their corresponding multiple attention real data.
12. The trajectory prediction device according to claim 10, wherein: In step (II), The trajectory prediction device enables the interpretation module to (i) generate multiple test latent features corresponding to each test frame by dimensionality reduction of the test image, multiple test metadata, and multiple test features corresponding to each test frame through the encoder of the autoencoder, and (ii) reconstruct the multiple test latent features corresponding to each test frame through the decoder of the autoencoder to generate multiple test interpretations corresponding to each test frame for multiple test factors that affect the test trajectory prediction.
13. The trajectory prediction device according to claim 10, wherein: In step (II), The trajectory prediction device causes the interpretation module (i) (i-1) to perform learning operations on multiple test images, multiple test metadata and multiple test features corresponding to each test frame to generate multiple test semantic segmentation images corresponding to each test frame, and (i-2) to identify instance-level regions of interest in the test semantic segmentation images by an autoencoder, and then (ii) to generate a test interpretation image corresponding to each test frame, and to mark multiple test factors that affect the test trajectory prediction with reference to the results of (i-2) above.
14. The trajectory prediction device according to claim 10, wherein: In step (II), The trajectory prediction device enables the interpretation module to perform learning operations on multiple test metadata corresponding to each test frame, and generate multiple test decision trees based on all the multiple test metadata associated with the test object in the multiple test images corresponding to each test frame.
15. The trajectory prediction device according to claim 14, wherein: In step (III), The trajectory prediction device causes the visualization module to reference multiple test decision trees and multiple test interpretations to output state information about at least one target object in each test frame that affects the test trajectory prediction.
16. The trajectory prediction device according to claim 10, wherein: In step (I), The trajectory prediction device inputs multiple test images and multiple test sensing information corresponding to each frame into the metadata recognition module, so that the metadata recognition module (1) detects the environmental information around the multiple devices through the perception module, and (2) detects the positioning information of the devices through the positioning and mapping module.
17. The trajectory prediction device according to claim 10, wherein: The test metadata corresponding to each frame includes at least multiple object bounding boxes corresponding to multiple test objects, multiple pose information of the device, and map information corresponding to the area where the device is located.
18. The trajectory prediction device according to claim 10, wherein: The trajectory prediction module includes a recurrent neural network, which consists of at least one of a long short-term memory algorithm and a long short-term memory-generative adversarial network algorithm.