System and method for predicting pedestrian safety information based on video

Abstract

The invention discloses a method and a device for predicting pedestrian safety information based on a video. Firstly, pedestrian trajectory prediction data based on video input is generated. Pedestrian behavior prediction data based on the video input is then generated. Potential risks of pedestrian safety are assessed based on pedestrian trajectory prediction data, pedestrian behavior prediction data, and surface classification data in video data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of Korean Patent Application No. 10-2023-0098385, filed on July 27, 2023, and Korean Patent Application No. 10-2024-0000781, filed on January 3, 2024, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to a system and method for predicting pedestrian safety information based on video. Background Art

[0004] In intelligent video analysis systems, object detection and tracking schemes have become basic building blocks, and image segmentation algorithms are frequently employed.

[0005] However, according to this technology, since the risk is evaluated after observation with a camera, it is not suitable for situations where proactive measures such as accident prevention through alarms are required.

[0006] Furthermore, when only the past and current locations of objects are used to construct a traffic accident alert system, in order to achieve a target accident prevention rate, the traffic accident alert system will be too sensitive, resulting in frequent false alarms.

[0007] Furthermore, most of the state-of-the-art trajectory prediction techniques only use coordinate data or visual feature data aggregated via average pooling.

[0008] In addition, since existing technologies related to pedestrian behavior prediction are all based on image data and vehicle sensors, etc., there is a problem of insufficient prediction accuracy. Summary of the invention

[0009] In order to solve the above-mentioned problems, the purpose of this disclosure is to propose a system and method for predicting pedestrian safety information based on video analysis, which can evaluate the future trajectory, behavior and potential risks of pedestrians before potential accidents occur. In the case of accident risks in the traffic environment, proactive actions can be taken through prediction to mitigate possible risks in the future.

[0010] However, the objects of the present disclosure are not limited to the above objects, and there may be other objects.

[0011] In order to solve the above problems, according to the first embodiment of the present disclosure, a method for predicting pedestrian safety information based on video includes: generating pedestrian trajectory prediction data based on video; generating pedestrian behavior prediction data based on video; and evaluating pedestrian safety risks based on trajectory prediction data, behavior prediction data and surface classification results of video input.

[0012] In a second embodiment of the present disclosure, a system for predicting pedestrian safety information based on video includes: a communication module configured to receive video captured by a camera installed at a predetermined location; a memory configured to store a program for assessing pedestrian risk based on the captured video; and a processor configured to: generate pedestrian trajectory prediction data based on the video, generate pedestrian behavior prediction data based on the video, and assess risk based on the trajectory prediction data, the behavior prediction data, and a surface classification result in the video input by running the program stored in the memory.

[0013] In a third embodiment of the present disclosure, a system for predicting pedestrian safety based on video includes: a communication module configured to receive a video captured by a camera installed at a predetermined position; a memory configured to store a program for assessing pedestrian risks based on the captured video; and a processor configured to: extract visual feature data for pedestrian trajectory prediction based on the video by running the program stored in the memory and generate pedestrian trajectory prediction data by inputting the visual feature data into a cyclic encoder and a decoder, extract multiple visual input feature data and non-visual input feature data from the video and generate pedestrian behavior prediction data by grouping the corresponding extracted input feature data and inputting them into corresponding processing modules and then concatenating the output results, and assess pedestrian semantic position data and potential risks based on the pedestrian trajectory prediction data, the pedestrian behavior prediction data and the surface classification data in the video.

[0014] In order to solve the above problem, according to another embodiment of the present disclosure, a computer program is cascaded with a hardware computer to run a method for predicting pedestrian safety information based on a video, and is stored in a computer-readable recording medium.

[0015] Other specific matters of the present disclosure are included in the detailed description and the accompanying drawings.

[0016] Unlike general traffic accident systems that are primarily used to respond to situations where an accident has already occurred, the embodiments of the present disclosure enable more effective pedestrian safety and accident prevention through pedestrian trajectory prediction, pedestrian behavior prediction, and risk assessment, by focusing on detecting and preventing accident risks before an accident occurs.

[0017] In addition, the current CCTV-based alarm system mainly generates alarms based on the location of pedestrians, so there is a problem of frequent false alarms. However, according to the embodiments of the present disclosure, since the alarm is generated by predicting the trajectory, location and risk, the false alarm rate is reduced, thereby improving the system reliability and reducing false alarms.

[0018] The effects of the present disclosure are not limited to the above-mentioned effects, and those skilled in the art will be able to clearly understand other unmentioned effects from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention is a block diagram of a system for predicting pedestrian safety information according to an embodiment of the present disclosure.

[0020] Figure 2 The diagram is a functional block diagram of a system for predicting pedestrian safety information according to an embodiment of the present disclosure.

[0021] Figure 3 The diagram schematically illustrates the operation of the system for predicting pedestrian safety information according to an embodiment of the present disclosure.

[0022] Figure 4 The flowchart is a method for predicting pedestrian safety information according to an embodiment of the present disclosure.

[0023] Figure 5 A diagram illustrating a process of generating pedestrian trajectory prediction data according to an embodiment of the present disclosure.

[0024] Figure 6 A diagram illustrating the structure of an attention visual feature module according to an embodiment of the present disclosure.

[0025] Figure 7 A diagram for explaining an overview of generating pedestrian behavior prediction data according to an embodiment of the present disclosure.

[0026] Figure 8 is a flowchart illustrating a method for generating pedestrian behavior prediction data according to an embodiment of the present disclosure.

[0027] Fig. 9 is a diagram illustrating various processing modules for executing a method for generating pedestrian behavior prediction data according to an embodiment of the present disclosure.

[0028] Fig.10 Detailed operations of various processing modules for executing a method for generating pedestrian behavior prediction data according to an embodiment of the present disclosure are illustrated.

[0029] Fig.11 is a flow chart illustrating a method for pedestrian risk assessment according to an embodiment of the present disclosure.

[0030] Fig.12 is a diagram illustrating a process of generating a ground area map according to an embodiment of the present disclosure.

[0031] Fig.13 is a diagram showing an example of a ground area map evaluated according to an embodiment of the present disclosure.

[0032] Fig.14 is a diagram illustrating a process of evaluating pedestrian semantic position data according to an embodiment of the present disclosure.

[0033] Fig.15 is a diagram illustrating the process of evaluating pedestrian footprint areas.

[0034] Fig.16 is a diagram showing an example of a result of evaluating pedestrian semantic position data according to an embodiment of the present disclosure.

[0035] Fig.17 is a flow chart illustrating a method for pedestrian risk assessment according to an embodiment of the present disclosure.

[0036] Fig.18A is a diagram illustrating a process of evaluating pedestrian risk according to an embodiment of the present disclosure.

[0037] Fig.18B is a diagram illustrating an evaluation network for evaluating pedestrian risk according to an embodiment of the present disclosure.

[0038] Fig.19 is a diagram showing trajectory prediction data in the PIE dataset.

[0039] Fig. 20 is a diagram showing trajectory prediction data in a data set of dangerous road walking behaviors of children in a child protection area according to an embodiment of the present disclosure.

[0040] Fig.21 is a diagram illustrating the qualitative analysis results of pedestrian behavior prediction data according to an embodiment of the present disclosure.

[0041] Fig. 22 is a diagram illustrating another qualitative analysis result of pedestrian behavior prediction data according to an embodiment of the present disclosure.

[0042] Fig.23 is a diagram showing an example of a feature fusion method in the related art.

[0043] Fig.24 is a diagram showing an example of a pedestrian risk assessment result according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] By referring to the embodiments that will be described in detail below with reference to the accompanying drawings, aspects and features of the present disclosure and methods for achieving these aspects and features will become apparent. However, the present disclosure is not limited to the embodiments disclosed below, but can be implemented in a variety of different forms. However, the embodiments are provided to improve the present disclosure and help those of ordinary skill in the art to fully understand the scope of the present disclosure, and the present disclosure is limited only by the scope of the attached claims.

[0045] The terms used in the specification are used to explain the embodiments, but are not intended to limit the present disclosure. In the specification, unless otherwise specified in the context, the singular form includes the plural form. In the specification, the terms "include" and / or "include" should be interpreted as not excluding the presence or addition of one or more other constituent elements in addition to the mentioned constituent elements. Throughout the specification, the same reference numerals are used to indicate the same constituent elements, and the term "and / or" includes all combinations of each mentioned constituent element and one or more thereof. The terms "first", "second", etc. are used to describe a variety of constituent elements, but these constituent elements should not be limited by these terms. The above terms are only used to distinguish the purpose of one constituent element from another constituent element. Therefore, the first constituent element to be mentioned below can be a second constituent element in the technical concept of the present disclosure.

[0046] In this specification, "pedestrian safety information" includes information about the degree of potential risk to pedestrian safety.

[0047] Unless otherwise defined, all terms (including technical and scientific terms) used in this specification can be used as the meanings commonly understood by those skilled in the art to which the present disclosure belongs. In addition, unless otherwise clearly and specifically defined, the terms defined in general dictionaries should not be idealized or over-interpreted.

[0048] The following will refer to Figures 1 to 3 A system for predicting pedestrian safety information based on video according to an embodiment of the present disclosure is described in detail.

[0049] Figure 1 is a block diagram of a system for predicting pedestrian safety information according to an embodiment of the present disclosure. Figure 2 is a diagram showing functional blocks of a system for predicting pedestrian safety information according to an embodiment of the present disclosure. Figure 3 Schematic diagram of the operation of the system for predicting pedestrian safety information according to an embodiment of the present disclosure.

[0050] The system 100 for predicting pedestrian safety information according to an embodiment of the present disclosure includes a communication module 110 , a memory 120 , and a processor 130 .

[0051] The communication module 110 receives the video captured by the camera (e.g. CCTV camera) installed at a predetermined position. The communication module 110 may include both a wired communication module and a wireless communication module. The wired communication module may be implemented by a power line communication device, a telephone line communication device, a cable trough (MoCA), Ethernet, IEEE1294, an integrated trough network, and an RS-485 control device. In addition, the wireless communication module may include a module for implementing functions such as wireless LAN (WLAN), Bluetooth, HDR WPAN, UWB, ZigBee, impulse radio, 60GHz WPAN, binary CDMA, wireless USB technology, and wireless HDMI technology, other fifth generation (5G) communications, long term evolution-advanced (LTE-A), long term evolution (LTE), and wireless fidelity (Wi-Fi).

[0052] The program for evaluating the risk of pedestrians based on video data is stored in the memory 120. Here, the memory 120 is a general term for a volatile memory device and a non-volatile memory device that can continuously retain the stored data even without power. For example, the memory 120 may include NAND flash memory (such as a compact flash (CF) card, a secure digital (SD) card, a memory stick, a solid state drive (SSD), and a micro SD card), a magnetic computer storage device (such as a hard disk drive (HDD)), and an optical disk drive (such as a CD-ROM and a DVD-ROM).

[0053] The processor 130 may control at least one other constituent element (eg, hardware or software constituent element) of the system 100 for predicting pedestrian safety information. The processor 130 may perform various data processing or operations by running software (eg, a program).

[0054] In addition, in an embodiment of the present disclosure, the processor 130 may use at least one of machine learning, neural network, or deep learning algorithm as an artificial intelligence algorithm for predicting pedestrian safety information, but the present invention is not limited thereto. As an example, at least one of machine learning, neural network, or deep learning algorithm may be used as an artificial intelligence algorithm, and examples of neural networks may include models such as convolutional neural networks (CNNs), deep neural networks (DNNs), and recurrent neural networks (RNNs).

[0055] In the embodiments of the present disclosure, artificial intelligence (AI) may be a technology that imitates human learning ability, reasoning ability, and perception ability and realizes these abilities through computers. Artificial intelligence may mainly include machine learning and technical elements that utilize machine learning. Machine learning may be an algorithm that extracts at least one learning data feature to classify input data. In addition, the technology that replicates the functions of cognition and judgment in the human brain by utilizing a machine learning algorithm may also be understood to belong to the category of artificial intelligence. For example, technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge expression, and operational control may all be included in the category of artificial intelligence.

[0056] In an embodiment of the present disclosure, an artificial neural network may be designed to implement a human brain structure on a computer, may simulate neurons of a human neural network, and may include a plurality of network nodes having weight values. A plurality of network nodes may have a connection relationship with each other by simulating the synaptic action of neurons, wherein neurons send and receive signals through synapses. In an artificial neural network, a plurality of network nodes may be located on layers with different depths, and data may be sent and received according to a convolutional connection relationship. The artificial neural network may be, for example, a convolutional neural network model, etc.

[0057] In the present disclosure, an artificial neural network is a model learned according to a predetermined machine learning method, and may be a model that determines the weight of at least one network node included in an unlearned model by machine learning. Machine learning may refer to computer software improving data processing capabilities by learning using data and data processing experience. An artificial neural network may be established by modeling the correlation between data, and the correlation may be represented by multiple parameters. An artificial neural network may derive the correlation between data by extracting and analyzing features from given data, and optimizing the parameters of an artificial neural network by repeating such a process may be referred to as machine learning. For example, an artificial neural network may learn the mapping (correlation) between input and output with respect to data given as an input / output pair. In addition, even when only input data is given, an artificial neural network may learn the relationship between given data by deriving the law between the data. In the present disclosure, "artificial neural network" may be used interchangeably with the term "artificial neural network model" or "artificial intelligence algorithm".

[0058] refer to Figure 2 According to an embodiment of the present disclosure, the system 100 for predicting pedestrian safety information includes a pedestrian trajectory prediction unit 210 , a pedestrian behavior prediction unit 220 and a pedestrian risk assessment unit 230 .

[0059] The pedestrian trajectory prediction unit 210 predicts the future position of the pedestrian based on the pedestrian's past trajectory and the video input.

[0060] The pedestrian behavior prediction unit 220 determines the intention of the pedestrian based on the video input. For example, the pedestrian behavior prediction unit 220 predicts whether the pedestrian will leave the sidewalk and cross the road.

[0061] The pedestrian risk assessment unit 230 assesses the final risk by synthesizing the future position via trajectory prediction, behavior prediction data, and ground classification results of the video.

[0062] like Figure 3 As shown, the embodiments of the present disclosure can predict the intention 320 and predicted trajectory position 330 of the pedestrian based on the past trajectory 310, can evaluate the risk based on the risk prediction 340, and can provide appropriate risk warnings 350 to road users (such as drivers), thereby preventing traffic accidents.

[0063] Below, reference Figure 4 to Figure 1 8 , the method performed by the system 100 for predicting pedestrian safety information based on video according to an embodiment of the present disclosure will be described in detail.

[0064] Figure 4 The flowchart is a method for predicting pedestrian safety information according to an embodiment of the present disclosure.

[0065] According to an embodiment of the present disclosure, a method for predicting pedestrian safety information includes: generating pedestrian trajectory prediction data based on video input (S410); generating pedestrian behavior prediction data based on video input (S420); and evaluating pedestrian safety risks based on trajectory prediction data, behavior prediction data and surface classification results in video input (S430).

[0066] Figure 5 is a diagram illustrating a process of generating pedestrian trajectory prediction data according to an embodiment of the present disclosure. Figure 6 is a diagram illustrating the structure of an attention visual feature module according to an embodiment of the present disclosure.

[0067] First, in an embodiment of the present disclosure, pedestrian trajectory prediction data is generated based on video input ( S410 ).

[0068] In known trajectory prediction methods, a feature vector is extracted by a recurrent neural network encoder, the past trajectory of the observed object is received as input, and then the future trajectory is predicted by using a recurrent neural network decoder. Most existing methods do not use video input during the entire trajectory prediction process. In addition, even when video input is used, visual features of the entire area (such as convolution, maximum pooling, and average pooling) are used to summarize the visual feature data in the entire observation area, so the performance of these methods is limited.

[0069] In order to solve these problems through the trajectory prediction method according to the embodiment of the present disclosure, an attention mechanism or a visual transformer can be used to provide visual features to the recurrent encoder and the recurrent decoder.

[0070] Specifically, the video input and the pedestrian's past trajectory data are input to the first attention visual feature module 510 based on the attention mechanism, and the bounding box corresponding to the pedestrian in the video input and the pedestrian's surrounding area in the video of the bounding box are provided to the loop encoder 520. The first attention visual feature module 510 can be called a pre-encoder AVF (attentive visual feature) module. The first attention visual feature module 510 is located in front of the loop encoder 520, and its output is input to the loop encoder 520. The first attention visual feature module 510 can provide visual feature data to the loop encoder 520. The first attention visual feature module 510 can receive an observed single video frame and pedestrian position data (bounding box) as its input. The first attention visual feature module 510 can extract visual information of pedestrians and pedestrian surrounding areas from the video (video frame).

[0071] The first attention visual feature module (pre-encoder AVF) 510 takes the video and the past trajectory of the pedestrian as input to provide visual feature data to the recurrent encoder 520. The encoded feature data from the recurrent encoder 520 is input to the second attention visual feature module (post-encoder AVF) 530. The recurrent decoder 540 takes the concatenation of the outputs of the recurrent encoder 520 and the second attention visual feature module (post-encoder AVF) 530 as input and generates pedestrian trajectory prediction data as described below.

[0072] In addition, the attention visual feature module 600 according to the embodiment of the present disclosure follows the structure of the visual transformer. In this case, the difference between the visual transformer and the attention visual feature module 600 lies in the implementation of the attention mechanism. In the case of the visual transformer, the mutual attention values ​​between all patch embeddings are calculated.

[0073] On the contrary, Figure 6 As shown, according to the attention visual feature module 600 of the embodiment of the present disclosure, in order to extract only the features of the area related to the input pedestrian trajectory prediction process, the key (K) and value (V) are obtained from the block embedding of the video. The attention visual feature module 600 calculates the query (Q) using the pedestrian bounding box and / or trajectory encoding feature data to reduce the computational complexity. Next, pedestrian behavior prediction data is generated based on the video input (S420). The following will refer to Figures 7 to 10The process of generating pedestrian behavior prediction data according to an embodiment of the present disclosure is described. The description about the attention visual feature module 600 can be applied to the attention visual feature module 510 or the second attention visual feature module 530 .

[0074] Figure 7 2 is a diagram illustrating an overview of generating pedestrian behavior prediction data according to an embodiment of the present disclosure.

[0075] First, the observation start time (experiment start frame) 710 is defined as the mth frame before the current time t. The observation data is collected from the (tm)th frame to the (t-1)th frame during the observation length m 720.

[0076] In addition, at the critical time 730, the prediction of the pedestrian's future behavior is performed, whether to cross the road. That is, based on the observation data of the past frame, it is predicted whether the pedestrian will leave the sidewalk and will be on the road. In addition, the prediction time 740 is defined as the time interval between the critical frame 730 and the crossing frame 750.

[0077] If the model predicts that the pedestrian leaves the sidewalk and crosses the road in key frame 730, and the pedestrian actually crosses the road in crossing frame 750, this corresponds to a case where the model predicts correctly. Conversely, if the pedestrian does not actually cross the road, this corresponds to a case where the model predicts incorrectly.

[0078] Figure 8 is a flowchart illustrating a method for generating pedestrian behavior prediction data according to an embodiment of the present disclosure.

[0079] According to an embodiment of the present disclosure, a method for generating pedestrian behavior prediction data includes: setting a video input within a predetermined time from the present to the past as an observation target video (S810); extracting a plurality of visual input feature data and non-visual input feature data from the observation target video (S820); grouping the plurality of visual input feature data and non-visual input feature data (S830); and generating pedestrian behavior prediction data by inputting the grouped feature data into a separate processing module and cascading the output results of the corresponding processing modules (S840).

[0080] Fig. 9 It is a diagram illustrating various processing modules used to implement the method for generating pedestrian behavior prediction data according to an embodiment of the present disclosure. Fig.10 Detailed operations of various processing modules for executing the method for generating pedestrian behavior prediction data according to an embodiment of the present disclosure are illustrated.

[0081] First, refer to Fig. 9According to an embodiment of the present disclosure, the mth video frame before the current time t is set as the observation target (S810). That is, the video frames from the (tm)th frame to the tth frame are set as the observation target video.

[0082] Next, multiple visual input feature data and non-visual input feature data are extracted from the observation target video ( S820 ), and the input feature data are grouped to suit their attributes ( S830 ).

[0083] Then, the grouped feature data is provided as input to the processing module 900. In this case, in an embodiment of the present disclosure, the processing module may include an observation module 910, a context module 920, and a convolution module 930.

[0084] Therefore, the non-visual features are acquired and processed by the observation module 910. Then, the visual features including the context features are input to and processed by the context module 920 including a convolutional neural network (CNN) and a recurrent neural network (RNN), and then processed by the convolution module 930 including a 3D convolutional network, a 3D maximum pooling and flattening layer, and a fully connected (FC) layer.

[0085] Thereafter, pedestrian behavior prediction data after the nth frame may be generated by cascading the output results of the three processing modules ( S840 ).

[0086] refer to Fig.10 According to an embodiment of the present disclosure, visual input feature data and non-visual input feature data are processed separately according to their characteristics.

[0087] In an embodiment of the present disclosure, the visual input feature data may include local context features (which are images having a size that is a predetermined multiple of the bounding box data), global context features (including video segmentation data), scene context features (including the entire video data), pedestrian regions obtained from the video input, and areas surrounding pedestrians after removing the pedestrian regions from the video input.

[0088] In one embodiment, the non-visual input feature data may include pedestrian posture feature data, pedestrian bounding box data, and vehicle dynamic data including vehicle speed, direction, steering angle, and angular velocity.

[0089] As described above, the visual input feature data and the non-visual input feature data are grouped and input to corresponding processing modules 900 .

[0090] To this end, in an embodiment of the present disclosure, pedestrian posture feature data, pedestrian bounding box data, and vehicle dynamic data can be generated as a first feature data set. In addition, local context features (i.e., images of predetermined sizes around pedestrian bounding boxes), global context features including video segmentation data, and scene context features including the entire video data can be used to generate a second feature data set. In addition, the pedestrian area obtained from the video input and the area around the pedestrian after removing the pedestrian area from the video input can be used to generate a third feature data set.

[0091] After the feature data are grouped, the first feature data group is input to the observation module 910 , the second feature data group is input to the context module 920 , and the third feature data group is input to the convolution module 930 .

[0092] First, the first feature data set processed by the observation module 910 corresponds to the non-visual input feature data. Pedestrian posture feature data, pedestrian bounding box data and vehicle dynamic data are input to the observation module 910.

[0093] The pedestrian posture feature data can be defined as the following formula 1.

[0094] [Formula 1]

[0095]

[0096] In Formula 1, "m" represents the observation length. For example, as pedestrian posture feature data, human posture feature data can be extracted from multiple coordinates, for example, 18 main key points (mouth, neck, shoulder, elbow, wrist, hip, knee, ankle, eye, ear, etc.), but the present invention is not limited to this. In this embodiment, these 18 key points are represented as 3D vectors, and each key point is composed of 2D coordinates in each time step.

[0097] The above pedestrian posture feature data is input to the first gated recurrent unit (GRU) module 1001, and the feature encoding of the pedestrian posture feature data is obtained from the first GRU module 1001. The GRU (Gated Recurrent Unit) module is a type of RNN (Recurrent Neural Network) module. In the present disclosure, GRU is used to process feature data, but other RNN (Recurrent Neural Network) technologies (such as LSTM) or CNN (Convolutional Neural Network) can also be applied to process feature data.

[0098] Next, the pedestrian bounding box can be defined as follows:

[0099] [Formula 2]

[0100]

[0101] In formula 2, It can be determined by the coordinates of the upper left corner ([(x1, y1)]) and the lower left corner ([(x2, y2)]) of each pedestrian. Since the bounding box data consists of four coordinates, L obs The dimension becomes m×4.

[0102] The pedestrian bounding box data may be concatenated with the feature encoding of the pedestrian posture feature data and may be input into the second GRU module 1002 , and the feature encoding of the pedestrian posture feature data included in the bounding box information may be obtained from the second GRU module 1002 .

[0103] Next, the vehicle speed information may be defined as in Equation 3 below.

[0104] [Formula 3]

[0105]

[0106] In formula 3, s i Represents the speed of the vehicle, and this input feature is obtained by installing sensors on the vehicle.

[0107] The vehicle dynamic data can be cascaded with the pedestrian posture feature data included in the bounding box data, and can be input into the third GRU module 1003, and the feature encoding of the posture feature data corresponding to the pedestrian included in the bounding box information and the vehicle dynamic data can be obtained from the third GRU module 1003.

[0108] Next, the second feature data group processed by the context module 920 includes three kinds of image inputs, i.e., local context features, global context features, and scene context features. Through the corresponding context inputs, image features are extracted by utilizing the VGG19 network pre-trained in ImageNet as the backbone of CNN. The features output from CNN 1021-1023 are processed by utilizing the first to the third GRU modules 1024-1026, and the outputs of the first to the third GRU modules are cascaded and then fused into one output. After the fused output is input to the fourth GRU module 1027, the attention mechanism 1028 is applied to the output of the fourth GRU module.

[0109] First, the local context feature can be defined as Equation 4 below.

[0110] [Formula 4]

[0111]

[0112] The images around pedestrians, including crosswalks, traffic lights, and traffic signs, are crucial for predicting pedestrian behavior. iis an image corresponding to an area with a predetermined size (e.g., 1.5 times larger than the pedestrian bounding box) around the pedestrian bounding box, and the RGB image is resized to 224 × 224 pixels. The feature vector is extracted as a vector of size (m, 512) and is reduced to a vector (m, 256) by processing through a maximum pooling layer with a 14 × 14 kernel.

[0113] The input of the global context feature is defined as follows:

[0114] [Formula 5]

[0115]

[0116] In formula 5, gc i It is obtained by extracting the semantic segmentation results using the DeepLabV3 model pre-trained in the Cityscapes dataset. The full scene and road surface can be classified by using the semantic map value, and the features are extracted and concatenated into a vector (m, 256) in a similar way to the local context features.

[0117] The input of scene context features can be defined as Equation 6 below.

[0118] [Formula 6]

[0119]

[0120] Here, sc i represents the entire image as well as the pedestrian and its surrounding area. In the same way as other context features, the entire image is resized to 224×224 pixels, processed by a max pooling layer with a 14x14 kernel, and fused with other context features.

[0121] Finally, the convolution module 930 receives the pedestrian area (local box) in the video input and the surrounding area (local surround) without the pedestrian area, and the two feature data are input to the 3D convolution network (C3D) 1031, 1032. Further, the output feature data of the 3D convolution network is sequentially reduced in dimension using the maximum pooling layer 1033, 1034 in the manner of 112→56→…→4. The image after dimensionality reduction is converted into a 1D image through the flattening layer 1036, 1037, and then passed to the fully connected layer 1038, 1039, so that the output feature data of the convolution module is generated by cascading the outputs of the fully connected layers 1038, 1039.

[0122] The pedestrian area can be defined as shown in Equation 7 below.

[0123] [Formula 7]

[0124]

[0125] In formula 7, b i Represents an image feature generated by cropping and padding the image feature (whose size is the same as the size of the bounding box), and then adjusting the longest dimension of the region of the cropped image to fit the desired output size. Thereafter, the final output of the feature data is passed to a 3D convolution layer 1031, followed by a maximum pooling layer 1033, and then flattened by a flattening layer 1036, and its size is adjusted to be able to be fused with other feature data.

[0126] The surrounding area from which the pedestrian area is removed may be defined as in Equation 8 below.

[0127] [Formula 8]

[0128]

[0129] In Formula 8, based on the local context feature, r i Represents an image cropped at a predetermined size around the bounding box (e.g., an area 1.5 times larger than the bounding box), but where the area corresponding to the bounding box is grayed out so that only visual features from the area around the bounding box are utilized. This enables visual information around the pedestrian area to be utilized, and the 3D convolution acts in the same way as in the case of local context feature data, so that future output values ​​are cascaded.

[0130] Subsequently, according to one embodiment of the present disclosure, the output of the first processing module 910 for processing the first feature data group is input to the first attention module 1040, and the output of the second processing module 920 for processing the second feature data group is input to the second attention module 1028. Further, the output result of the second attention module is cascaded with the third feature data group and then input to the third attention module 1042. Subsequently, the outputs of the first to third attention modules are cascaded and then passed to the fourth attention module 1044, so that pedestrian behavior prediction data can be calculated.

[0131] In addition, in the embodiment of the present disclosure, in order to consider the temporal context of the input features, a gated recurrent unit (GRU) module is used, which is known to have a simpler and more efficient structure than the long short-term memory (LSTM). As the cyclic behavior of the GRU equation, the variables of the jth level of the stack are defined as follows in Equation 9.

[0132] [Formula 9]

[0133]

[0134] Here, σ(·) is the logistic sigmoid function, and represents the input features at time step t. W is the weight between two units, and The reset gate and update gate corresponding to time step t. and denote the hidden states of the past time step and the current time step respectively.

[0135] Next, since the attention mechanism is used to extract features from a specific part of the video input, it is more suitable for extracting visual feature data related to a certain pedestrian bounding box. The output vector of the attention module can be defined as follows:

[0136] [Formula 10]

[0137] β attention =tanh(W c [h c :h m ])

[0138]

[0139] Here, W c is the weight matrix, m represents the observation time, h m represents the last hidden state of the encoder, and h c is the sum of all hidden states that have received attention weights. is the previous hidden state of the encoder, and α t is the attention weight vector. In this case, the attention weight vector can be defined as follows:

[0140] [Formula 11]

[0141]

[0142] here, is a content-based function, W p represents the learnable weight matrix.

[0143] In addition, according to an embodiment of the present disclosure, pedestrian behavior, that is, whether a pedestrian is crossing the road, can also be automatically annotated for the entire frame of the video input based on the pedestrian risk assessment unit 230. That is, according to an embodiment of the present disclosure, the automatic annotation method can be applied to the data set to enhance and expand the data set.

[0144] The pedestrian crossing annotations included in the existing dataset are not labeled for all frames, so the limitation is that the training of pedestrian behavior prediction can only be performed on the annotated parts. That is, if there are additional training samples, the prediction accuracy may be improved.

[0145] In order to obtain additional training samples, according to an embodiment of the present disclosure, an automatic labeling method capable of generating training samples for pedestrian behavior prediction is applied. That is, by using the pedestrian risk assessment unit 230, the pedestrian in the video is classified as being on the sidewalk or on the roadway. If the pedestrian is on the sidewalk, the pedestrian is labeled as a safe pedestrian, and if the pedestrian is on the roadway, the pedestrian is labeled as an unsafe pedestrian. Such labels can be added to the training samples for use in the learning process of pedestrian behavior prediction.

[0146] Thus, according to the embodiments of the present disclosure, a large amount of behavior prediction data in a video can be quickly annotated. In addition, since the annotation is based on the sidewalk and the roadway, rather than on human subjective judgment, the consistency of the annotation can be maintained. In addition, since fast annotation can be performed even for new videos, it is easy to improve the performance of pedestrian behavior prediction.

[0147] Next, the potential risk of the pedestrian is assessed based on the trajectory prediction data, behavior prediction data and ground classification results corresponding to the video input (S430). Figures 11 to 19 To describe the process of assessing the potential risk of pedestrians according to an embodiment of the present disclosure.

[0148] Fig.11 is a flowchart illustrating a method for pedestrian risk assessment according to a first embodiment of the present disclosure.

[0149] The method for pedestrian risk assessment according to an embodiment of the present disclosure includes: receiving a video captured by a CCTV camera installed at a predetermined location (S1110); generating a bounding box for a pedestrian present in the video (S1120); generating a segmentation map by semantically segmenting the video (S1130); generating a ground area map by identifying a ground area using the segmentation map (S1140); and evaluating semantic position information of the pedestrian based on the bounding box and the ground area map (S1150).

[0150] First, a video input is obtained by a CCTV camera installed at a predetermined location where pedestrians pass (such as a crosswalk or a child protection area) (S1110). Of course, real-time streaming video frames or pre-stored video files can be used.

[0151] Next, a bounding box for a pedestrian present in the video is generated (S1120). In this case, a certain technique may be applied to the method of generating a bounding box in a video of a person present in a scene. For example, a deep learning-based object detection module (You Only Look Once (YoLO), Single Shot MultiBoxDetector (SSD), Hough transform, and contour-based detection techniques may be applied.

[0152] Next, a segmentation map is generated by semantically segmenting the video (S1130). For reference, semantic segmentation is the process of classifying which object category each pixel in a video frame belongs to. In this case, the segmentation map can be represented as a probability map for each of the c categories. In step S1130, semantic segmentation is performed for the target object including areas related to the road surface (e.g., roads, crosswalks, and sidewalks).

[0153] Next, a ground area map is generated by identifying the ground area using the segmentation map (S1140). In step S1140, an area related to the ground surface is automatically identified by using a given segmentation map.

[0154] Fig.12 is a diagram illustrating a process of generating a ground area map according to an embodiment of the present disclosure. Fig.13 is a diagram illustrating an example of a ground area map according to an embodiment of the present disclosure.

[0155] Usually, in a single frame, the road surface observed by the camera is covered by various objects such as people and cars, making it difficult to accurately divide the ground area to identify the semantic position of pedestrians. Therefore, in the embodiments of the present disclosure, in order to solve this problem, the ground area does not change much over time when the CCTV camera is fixedly installed, and the ground area can be extracted by accumulating the video segmentation results, and the object area information is excluded.

[0156] According to one embodiment, in order to generate a ground area map, a ground area mask is generated to temporally accumulate the probability of the class of interest (ground area) (S1210), wherein pixels belonging to the ground area are set to true. In this case, the ground area mask can be defined as in the following equation 12.

[0157] [Formula 12]

[0158]

[0159] In the above formula, x represents the pixel coordinate, P c,i is the segmentation map representing the probability of category c in the i-th frame, C _groundis the class set of ground areas of interest (roads, crosswalks, and sidewalks).

[0160] Next, by combining the ground region mask and the segmentation map, the cumulative probability of the ground region over time is calculated using the following equation 13 (S1220).

[0161] [Formula 13]

[0162]

[0163] In Equation 13, ° is an element-wise multiplication operator.

[0164] Next, a ground area map indicating to which ground area each pixel belongs is generated based on the cumulative probability ( S1230 ).

[0165] In the presently disclosed embodiment, a process of normalizing the cumulative probability may be additionally performed, and for the normalization, the number of pixels of the probability that each pixel is accumulated to the i-th frame is calculated by the following Equation 14.

[0166] [Formula 14]

[0167] N i (x) = N i-1 (x)+M i-1 (x)

[0168] Furthermore, a ground area map indicating which ground area each pixel belongs to can be calculated by the following formula 15.

[0169] [Formula 15]

[0170]

[0171] Fig.13 An example of a ground area map evaluated by the above process is shown in FIG, whereby the ground area can be identified from the video segmentation result even if the ground area is temporarily covered by a moving object. Fig.13 , the left column represents the input video (1310), the middle column represents the video segmentation results (1320), and the right column represents the road area assessment results (1330).

[0172] Reference again Fig.11 , then the semantic position information of the pedestrian is evaluated based on the bounding box and the ground area map (S1150).

[0173] Fig.14 is a diagram illustrating a process of evaluating pedestrian semantic position information according to an embodiment of the present disclosure. Fig.15 is a diagram illustrating a process of evaluating a pedestrian's foot region. Fig.16is a diagram illustrating an example of a result of evaluating pedestrian semantic position information according to an embodiment of the present disclosure.

[0174] First, in order to identify which ground area the pedestrian is actually standing on, the pedestrian's foot area is evaluated from the bounding box (S1410). Fig.15 The size information 1510 of the bounding box shown evaluates a bottom area 1520 having a predetermined ratio as a foot area of ​​the pedestrian. In this case, the predetermined ratio may be set to 10%, but the present invention is not limited thereto.

[0175] Next, a category corresponding to the most pixels on the ground area map contacting the foot area of ​​the pedestrian may be classified as semantic position information of the pedestrian ( S1420 ).

[0176] refer to Fig.16 , the semantic location information of the pedestrian may be separated and distinguished into the road 1610 , the crosswalk 1620 , and the sidewalk, and may be expressed respectively by colors and texts that are distinguished from each other.

[0177] Fig.17 is a flow chart showing a method for pedestrian risk assessment according to a second embodiment of the present disclosure. In this case, another embodiment of the present disclosure includes some of the same technical features as the first embodiment, so repeated content will be omitted as much as possible.

[0178] In addition, although the first embodiment of the present disclosure is to evaluate the semantic position information of pedestrians, and the second embodiment of the present disclosure is to evaluate the risk of pedestrians, they may not be performed separately, but may be additionally applied to each other's embodiments. This means that the pedestrian risk assessment may be additionally performed after the semantic position assessment of pedestrians is evaluated in the first embodiment.

[0179] First, if a video captured by a CCTV camera installed at a predetermined position is input ( S1710 ), a bounding box for a pedestrian existing in the video is generated ( S1720 ).

[0180] Next, a segmentation map is generated by semantically segmenting the video input ( S1730 ), and a ground area map is generated by identifying the ground area using the segmentation map ( S1740 ).

[0181] Next, the surrounding area and bounding box area of ​​the pedestrian in the video input are extracted from the video (S1750). Further, the pedestrian risk is evaluated based on the surrounding area and ground area map in the video (S1760).

[0182] Fig.18A is a diagram illustrating a process of evaluating pedestrian risk according to an embodiment of the present disclosure. Fig.18Bis a diagram illustrating an evaluation network for evaluating pedestrian risk according to an embodiment of the present disclosure.

[0183] In the present disclosure, the area around the pedestrian in the video is extracted from the video frame (S1812), and the ground area map obtained in the previous step (S1814) is used to evaluate whether the pedestrian is in a potentially dangerous position (road or crosswalk) or in a safe area (sidewalk). In this process, according to the embodiment of the present disclosure, a pedestrian is designed. Fig.18B The evaluation network shown is used to effectively fuse video frames and ground truth maps.

[0184] First, according to an embodiment of the present disclosure, a first feature map including visual features of the surrounding environment of a pedestrian may be extracted by inputting a video frame and a bounding box into an image encoder ( S1813 ).

[0185] Specifically, in order to extract the area around the pedestrian from the video input, a cropped video is generated by cropping an area of ​​a predetermined size (eg, an area twice as large as the pedestrian bounding box) surrounding the pedestrian bounding box ( S1811 ).

[0186] Next, the area corresponding to the bounding box is removed from the first cropped video (S1812). In the case of FIG. 18 , the extraction of the pedestrian visual information is excluded by filling the bounding box area of ​​the original pedestrian with gray.

[0187] Next, the first feature map including visual features for the area around the pedestrian is extracted by inputting the video with the area corresponding to the bounding box removed into a pre-trained visual feature extraction network (S1813). As an example, the first feature map F is extracted from the output of the last convolutional layer of VGG19 by inputting the pedestrian surrounding image into VGG19 pre-trained using the ImageNet dataset. img The first feature map includes a visual feature map of the area surrounding the pedestrian.

[0188] Furthermore, according to an embodiment of the present disclosure, a second feature map including ground features for pedestrians and pedestrian surroundings may be extracted by inputting the video, the bounding box, and the ground area map into a map encoder ( S1815 ).

[0189] Specifically, a second cropped video is generated by converting a ground area map into an RGB video and then cropping an area that is the same as that of the first cropped video (S1814). The ground area map includes ground information for determining on which ground the pedestrian is standing. In order to extract such information, the ground area map is converted into an image. In addition, the converted image is cropped according to the first video and resized to have a size of 224×224 pixels.

[0190] Next, a second feature map including ground features for pedestrians and pedestrian surroundings is extracted by inputting the second cropped video into a pre-trained road area feature extraction network (S1815). In this process, according to an embodiment of the present disclosure, an example of using atrous convolution to extract wider spatial features without downsampling is used. For the ground area map, three 3×3 atrous convolutions are applied, each with 32, 64, and 128 filters and expansion rates of 2, 4, and 8. In addition, the second feature map F is extracted by applying 32×32 maximum pooling to the output of the last layer. map The second feature map includes ground feature data of pedestrians and surrounding environment.

[0191] Next, in order to fuse features and maintain 2D spatial information present in the first feature map and the second feature map, the first and second feature maps are cascaded in the channel direction (S1816), and the convolution layer has a 3×3 kernel size and 512 filters (S1817).

[0192] Next, to convert the fused features into a feature vector, a flatten layer and a fully connected layer with 512 units are applied (S1818 and S1819).

[0193] Next, in order to perform learning to focus on the part of the generated feature vector that is helpful for classification, 8 heads are provided, each of which applies 32D multi-head self-attention (S1820).

[0194] Finally, by applying a fully connected layer with one unit, the pedestrian risk (probability of dangerous situation) is constructed as the final output (S1821).

[0195] In addition, in the above description, according to the implementation example of the present disclosure, steps S1110 to S1821 can be further subdivided into additional steps or can be combined into fewer steps. In addition, if necessary, some steps can be omitted, or the order between the steps can be changed. In addition, Figure 1 Content and Figure 2 to Figure 1 The contents of 8 can be applied to each other.

[0196] Below, reference Figures 19 to 24 , the effects of the system and method for predicting pedestrian safety information according to an embodiment of the present disclosure will be described in detail through the test results of the system and method.

[0197] Fig.19 is a diagram showing trajectory prediction data in the Pedestrian Intention Estimation (PIE) dataset. Fig. 20 1 is a diagram showing trajectory prediction data in a data set of dangerous road walking behaviors of children in a child protection zone.

[0198] A quantitative comparison of the pedestrian trajectory prediction performance of the disclosed embodiments is shown in Tables 1 and 2. The Joint Attention for Autonomous Driving (JAAD) and PIE datasets used for comparison in Table 1 include videos captured by on-board cameras and corresponding pedestrian trajectory annotations. On the other hand, the National Information Society Agency (NIA) dataset (videos of dangerous behaviors of children walking on roads in child protection areas) used in the comparative study in Table 2 contains videos obtained by fixed CCTV cameras in traffic scenes. If multiple future trajectory candidates are predicted from the performance, they are respectively represented by the suffix "*". It can be determined that the trajectory prediction method proposed in the performance comparison of the two tables shows the effectiveness of the embodiment compared with the existing pedestrian trajectory prediction methods.

[0199] [Table 1]

[0200]

[0201] [Table 2]

[0202]

[0203] Fig.19 and Fig. 20 The experimental results of qualitative comparison of predicted trajectories according to the embodiments of the present disclosure are shown. It is confirmed that pedestrian trajectory prediction through visual attention features is more accurate, and even in the case of multi-trajectory prediction, more reasonable pedestrian trajectory prediction results can be obtained.

[0204] exist Fig.19 and Fig. 20 , the first row 1910 and 2010 represent single trajectory prediction data using BiTraP-D, and the second row 1920 and 2020 represent single trajectory prediction data using an embodiment of the present disclosure. In addition, the third row 1930 and 2030 represent multi-trajectory prediction data in BiTraP-NP, and the fourth row 1940 and 2040 represent multi-trajectory prediction data in an embodiment of the present disclosure. In each figure, the solid line represents the pedestrian trajectory prediction data, and the dotted line represents the actual pedestrian trajectory.

[0205] Fig.21 is a diagram illustrating the qualitative analysis results of pedestrian behavior prediction data according to an embodiment of the present disclosure. Fig. 22 is a diagram illustrating another qualitative analysis result of pedestrian behavior prediction data according to an embodiment of the present disclosure.

[0206] The pedestrian behavior prediction accuracy (ACC) is determined based on the proportion of correctly predicted samples, and this follows the following Equation 16.

[0207] [Formula 16]

[0208]

[0209] In Equation 16, "TP" represents true positive, "TN" represents true negative, "FP" represents false positive, and "FN" represents false negative. "TP" represents the situation where the model predicts a pedestrian crossing the road and it actually occurs. "TN" represents the situation where the model predicts that the pedestrian does not cross the road and it actually occurs. "FP" represents the situation where the model predicts a pedestrian crossing the road but it actually does not occur. "FN" represents the situation where the model predicts that the pedestrian does not cross the road but the pedestrian actually crosses the road.

[0210] Next, the comparison results of the pedestrian behavior prediction performance obtained by the system and method proposed in the embodiments of the present disclosure are described. Representatively, in the PIE dataset, which is a dataset for pedestrian behavior prediction, the performance comparison results between the benchmark model in the present disclosure and existing different benchmark models are shown in Table 3 below.

[0211] [Table 3]

[0212]

[0213] The comparison metrics used in this comparison are accuracy (ACC), area under the curve (AUC), and F1-score. ACC is the prediction accuracy, and AUC is the area under the receiver operating characteristic (ROC) curve, and is defined as in Equation 17 below.

[0214] [Equation 17]

[0215]

[0216] Here, f is a predictor that returns the probability of the intention to cross the road, and 1[f(t0)<f(t1)] represents an indicator function that returns 1 when f(t0)<f(t1) and 0 otherwise. D 0 is the negative sample set, and D 1 is the positive sample set. If the AUC is high, it indicates that the model can effectively distinguish different classes.

[0217] The F1-score is the harmonic mean of precision and recall, and is defined as in Equation 18 below.

[0218] [Equation 18]

[0219]

[0220] Furthermore, in Table 1 above, P represents pedestrian pose feature data, L represents pedestrian bounding box information, and S represents vehicle speed information. Further, LC represents local context features, GC represents global context features, SC represents scene context features, B represents the pedestrian area, and R represents the area around the pedestrian in the video.

[0221] Compared with other existing baseline models, the embodiment of the present disclosure shows the biggest difference in that it uses both VGG and 3D convolutional network (C3D) as visual encoders for image features, and it is a network that can efficiently process most input features.

[0222] Existing methods 1 to 3 (SingleRNN, MultiRNN, and SFRNN) have something in common in using VGG as the visual encoder.

[0223] Existing method 1 (SingleRNN) is a method that concatenates bounding box information and ego vehicle speed information into one vector and shows performance of 0.76 / 0.64 / 0.45 for the lowest value of accuracy / AUC / F1. Existing method 2 (MultiRNN) is a method that outputs prediction uncertainty through Bayesian modeling and shows relatively good performance of 0.86 / 0.80 / 0.73 by using only four input features. Existing method 3 (SFRNN) is a method that uses the hidden state of the lower RNN as the input of the upper RNN.

[0224] Existing Method 4 Probabilistic Control Performance Assessment (PCPA) and Existing Method 5 (MCIP) extract result values ​​for C3D, I3D and VGG with visual encoder.

[0225] Existing method 4 (PCPA) shows the lowest performance of 0.86 / 0.85 / 0.77 when using C3D as the visual encoder, and the highest performance of 0.87 / 0.86 / 0.79 when encoding through the inflated 3D network (I3D). Existing method 5 (MCIP) uses a total of 5 inputs, including the input features used in existing method 4 (PCPA) and an additional segmentation map input. In addition, unlike existing method 4 (PCPA), it shows the lowest performance of 0.85 / 0.81 / 0.74 when encoding with I3D, and the highest performance of 0.89 / 0.87 / 0.81 when encoding with VGG.

[0226] In contrast, in the embodiment of the present disclosure, the visual input feature data is divided into two groups, one group is encoded by VGG and the remaining group is encoded by C3D. In addition, the network is designed to process efficiently by using 8 input features, which is the most inputs compared to existing methods. Compared with existing baseline methods, the performance of 0.91 / 0.89 / 0.84 is the highest, demonstrating the advantages of the present disclosure.

[0227] Table 4 below shows the results of the impact of the corresponding inputs on the overall performance by comparing the performance in the case of removing the corresponding input features one by one in one embodiment of the present disclosure. In this case, the input of the bounding box information has the greatest impact on the accuracy, showing a low performance of 0.063 / 0.043 / 0.084. The input with the least impact on the performance is the scene context feature, showing a performance drop of only about 0.010 / 0.004 / 0.013. Since the entire area of ​​the image is used instead of a small part of the image, it is determined that this impact will not have a significant impact on the prediction performance. Through this experiment, it can be concluded that all the input features listed in Table 4 contribute to the accuracy of pedestrian behavior prediction.

[0228] [Table 4]

[0229]

[0230] Further, according to an embodiment of the present disclosure, the visual input is divided into two groups according to the features and encoded by VGG and C3D. Table 5 shows that the performance is reduced when the input of the C3D group is encoded by VGG. It can be seen that the lowest reduction corresponds to a reduction of about 0.02 / 0.02 / 0.03. Therefore, it is shown that applying visual encoding separately by dividing each group proposed in the present disclosure is an effective method to improve prediction performance.

[0231] [Table 5]

[0232]

[0233]

[0234] Table 6 shows the results of the performance comparison between the model applied to the system proposed in the embodiment of the present disclosure and the existing model when the prediction time changes from 1 second to 4 seconds. In all experimental settings, the model according to the present disclosure consistently shows better performance than the existing model. The largest difference in performance occurs after 1 second prediction, where the accuracy / AUF / F1 score of the model according to the present disclosure is improved to 0.02 / 0.02 / 0.03 compared with the existing model.

[0235] [Table 6]

[0236]

[0237] Further, according to an embodiment of the present disclosure, by utilizing automatic annotation technology, annotations included in existing data sets are added, expanded and used for learning. Table 7 shows the experimental results, in which the simplicity of the automatic annotation technology according to an embodiment of the present disclosure can be seen, and the performance is compared for the manually corrected version and the data set (automatic annotation) version in which only the automatic annotation data set and the manually annotated data set are included. As shown in Table 7, compared with the manually corrected data set, the automatic annotation technology according to the present disclosure shows a negligible performance drop of about 0.01 / 0.00 / 0.01 based on accuracy / AUC / F1 scores. It can therefore be concluded that sufficient performance can be achieved by automatically annotating data sets.

[0238] [Table 7]

[0239] Dataset ACC AUC F1 Accuracy Recall Automatic labeling 0.88 0.85 0.92 0.95 0.90 Manual correction 0.89 0.85 0.93 0.95 0.91

[0240] Table 8 shows the performance comparison as the number of automatically labeled data increases. Table 8 shows the changing trend of the performance indicators as the number of automatically labeled data increases from 60 samples to 140 samples. When the number of training sets is 60, the accuracy / AUC / F1 scores are 0.78 / 0.76 / 0.85, respectively. As the number of training sets gradually increases by using more automatically labeled samples, the performance gradually improves, so that when the number of training sets is 140, the accuracy / AUC / F1 scores reach 0.88 / 0.85 / 0.92, which is the highest performance. From this experiment, it can be concluded with certainty that automatically labeled data effectively improves the performance of pedestrian behavior prediction.

[0241] [Table 8]

[0242] The number of training sets ACC AUC F1 Accuracy Recall 60 0.78 0.76 0.85 0.92 0.79 80 0.79 0.80 0.86 0.94 0.79 100 0.83 0.85 0.89 0.96 0.82 120 0.86 0.84 0.91 0.95 0.87 140 0.88 0.85 0.92 0.95 0.90

[0243] Fig.21 A quantitative analysis of the results of executing the model proposed in the present disclosure in the National Information Society Agency (NIA) dataset (videos of children's dangerous road walking behaviors in child protection areas) is shown. The solid-line box indicates that the pedestrian is predicted to be safe, while the dashed-line box indicates that the pedestrian is predicted to be accident-prone. In case 1 (2110), when the student who is crossing the road is within the crosswalk, the student is predicted to be accident-prone, but when the student is close to the sidewalk, the student is predicted to be safe. In case 2 (2120), two students are predicted to be accident-prone when entering the roadway. Finally, in case 3 (2130), when a student runs from the child protection area to the roadway, the student is predicted to be accident-prone, and when the student crosses the sidewalk, the student is predicted to be safe. Therefore, according to the present disclosure, it can be determined that pedestrian safety hazards are correctly predicted in advance.

[0244] Fig. 22A quantitative analysis of the results of executing the proposed model in the PIE dataset (videos of pedestrian behavior captured in vehicles) is shown. Fig. 20 Conversely, dashed boxes represent pedestrians predicted to be accident-prone, while solid boxes represent pedestrians predicted to be safe. In case 1 (2210), a pedestrian who intends to enter the roadway is continuously predicted to be accident-prone. In case 2 (2220), a pedestrian who enters the crosswalk is first predicted to be accident-prone, but as the pedestrian gradually stops, the predicted state changes to safe. In case 3 (2230), as the pedestrian in the roadway walks onto the sidewalk, the pedestrian is continuously predicted to be safe. Therefore, it can be determined that even for videos captured by a camera mounted on a moving vehicle, the safety hazard of a pedestrian can be correctly predicted in advance.

[0245] Fig.23 A diagram showing an example of a feature fusion method in the prior art. Fig.23 FIG. 2 shows an example of pedestrian risk assessment results in an embodiment of the present disclosure. CAT1D (2310) converts F img and F map Cascade, and then pass them through the fully connected (FC) layer to get the output value. CAT1D_SA (2320) img and Fm ap Cascade them, pass them through the FC layer, and then pass them through the attention layer and FC layer in turn to get the output value. CAT2D (2330) img and F map Concatenate them, pass them through the Conv2D layer, and then pass them through the Flatten layer and the FC layer in sequence to get the output value.

[0246] In order to evaluate the quantitative performance according to the embodiments of the present disclosure, 200 video clips (30 seconds each) were used in the "Children's Road Walking Dangerous Behavior Video in the Child Protection Zone" provided by the AI ​​Center. From the corresponding videos, 71,898 dangerous situation ROIs and 68,181 safe situation ROIs were generated and evaluated by dividing the video clips into 100 training sets and 100 test sets. For performance indicators, accuracy (ACC), precision, recall, F1 score, and area under the ROC curve (AUC) were used.

[0247] In this case, accuracy, precision, and recall are defined as in Equations 19 to 21 below.

[0248] [Formula 19]

[0249]

[0250] [Formula 20]

[0251]

[0252] [Formula 21]

[0253]

[0254] In this case, TP represents the number of true positive samples, TN represents the number of true negative samples, FP represents the number of false positive samples, and FN represents the number of false negative samples.

[0255] The F1 score is the harmonic mean of precision and recall and provides a balanced assessment of model performance.

[0256] [Formula 22]

[0257]

[0258] AUC is the area at the bottom of the ROC curve and is defined as follows:

[0259] [Formula 23]

[0260]

[0261] Here, p represents the safety status probability of the corresponding sample, 1[f(·)] is a function that is 1 when the built-in function is true and 0 when the built-in function is false, S 0 is the negative sample set, S 1 is the positive sample set.

[0262] Through the embodiments of the present disclosure, Fig.23 The feature fusion method shown is implemented to compare the accuracy of judging the dangerous condition of pedestrians by fusing multiple information features with the accuracy according to other conventional methods often used for fusion.

[0263] In this case, by img 、F map Apply 2D global average pooling to obtain a 1D feature vector f img 、f map .

[0264] As shown in the experimental results of Table 9, the semantic feature fusion of the ground is better than the fusion performance obtained by using only video frames (IMG_ONLY). In addition, it is found that using 2D feature maps instead of 1D feature vectors is crucial for classification accuracy because 2D feature maps retain the spatial information of the scene. Finally, it can be concluded that the feature fusion method according to the embodiment of the present disclosure shows the best performance.

[0265] [Table 9]

[0266]

[0267] exist Fig.24 In , the pedestrian risk assessment is represented by a bounding box, the solid line represents a safe state (2410), and the dashed line represents a dangerous state (2420). Fig.24 In the embodiment, the paired videos represent the transition of the potential risk of the pedestrian, and it can be concluded that the pedestrian risk assessment is performed correctly according to the embodiment of the present disclosure.

[0268] The above-described embodiments of the present disclosure may be implemented as a program (or application program) to be executed in conjunction with a hardware computer and may be stored in a medium.

[0269] In order to make the computer read the above-mentioned program to run the above-mentioned method implemented as a program, the program may include a code written by a computer language (such as C, C++, JAVA, Ruby and machine language), and the code may be read by the processor (CPU) of the computer via the device interface of the computer. This code may include a function code related to the function required for defining the operation of the above-mentioned method, and may also include a control code related to the operation process required for the processor of the computer to run the above-mentioned function according to a specific program. In addition, this code may also include a memory reference-related code about the additional information or medium required for the processor of the computer to run the above-mentioned function, which position (address) is referenced in the computer's internal or external memory. In addition, in the case where the processor of the computer needs to communicate with any other remote computer or server to run the above-mentioned function, the code may also include a communication code about how to use the communication module of the computer to communicate with any other remote computer or server, or which information or media are to be sent / received during the communication process.

[0270] Storage media refers to media that semi-permanently stores data and can be read by a device, rather than media that temporarily stores data, such as registers, caches, or memories. Specific examples of storage media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices, but are not limited thereto. That is, the program can be stored in various recording media on various servers that can be accessed by the computer, or in various recording media on the user's computer. In addition, the medium can be distributed in computer systems connected by a network, and can store codes that can be read by computers in a distributed manner.

[0271] The above explanation of the present disclosure is for illustrative purposes, and those skilled in the art to which the present disclosure belongs can understand that the present disclosure can be easily modified in other specific forms without changing the technical ideas or basic features of the present disclosure. Therefore, it should be understood that the above embodiments are illustrative in all aspects, rather than restrictive. For example, each component element explained as a single type can be distributed and executed, and in the same way, the component elements explained as distribution can be executed in a combined form.

[0272] The scope of the present disclosure is defined by the appended claims described later rather than the detailed description above, and all changes or modifications derived from the meaning, scope, and equivalent concepts of the claims should be construed as being included in the scope of the present disclosure.

Claims

1. A method for predicting pedestrian safety information based on video, performed by a computer, the method comprising the following steps: Generate pedestrian trajectory prediction data based on video input; generating pedestrian behavior prediction data based on the video input; as well as Pedestrian safety risk is assessed based on the trajectory prediction data, the behavior prediction data, and the surface classification data of the video input.

2. The method according to claim 1, wherein: The steps of generating pedestrian trajectory prediction data based on video input include the following steps: Based on the attention mechanism, by inputting the video input and the past trajectory data of the pedestrian corresponding to the video input into a first attention visual feature module, providing the past trajectory data and the visual feature data of the pedestrian to the recurrent encoder; and By inputting the video data and the past state feature vector extracted from the recurrent encoder into a second attention visual feature module, the visual feature data for future pedestrian trajectory prediction is output.

3. The method according to claim 2, wherein: The steps of generating pedestrian trajectory prediction data based on video input include the following steps: By inputting the pedestrian's past trajectory and visual feature data into the recurrent encoder, a pedestrian's past state feature vector is output; and The pedestrian trajectory prediction data is generated by inputting the visual feature data into the loop decoder.

4. The method according to claim 1, wherein: The steps of generating pedestrian behavior prediction data based on video input include the following steps: The video input from the present to a predetermined time in the past is set to observe the target video; Extracting a plurality of visual input feature data and non-visual input feature data from the observed target video; grouping the plurality of visual input feature data and the non-visual input feature data; and The grouped feature data are input into a separate processing module, and the pedestrian behavior prediction data is generated based on the output results of the processing module.

5. The method according to claim 4, wherein: The step of extracting the plurality of visual input feature data and the non-visual input feature data from the observed target video and the vehicle dynamic sensor comprises the following steps: Extracting pedestrian posture feature data, pedestrian bounding box and vehicle dynamic data as the non-visual input feature data; and Local context features, global context features, scene context features, a pedestrian area obtained from the video input, and an area around pedestrians after the pedestrian area in the video input is removed are extracted as the visual input feature data, the local context features are images with a size that is a predetermined multiple of the bounding box, the global context features include video segmentation data, and the scene context features include the entire area in the video input.

6. The method according to claim 5, wherein: The step of grouping the plurality of visual input feature data and the non-visual input feature data comprises the following steps: Generate a first feature data set, the first feature data set comprising the pedestrian posture feature data, the pedestrian bounding box and the vehicle dynamic data; generating a second feature data set, the second feature data set comprising: the local context feature, the global context feature comprising the video segmentation data, and the scene context feature comprising the entire area of ​​the video input, the local context feature being an image having a predetermined size around the bounding box; and A third feature data set is generated, the third feature data set including the pedestrian region obtained from the video input and the pedestrian surrounding region in the video after the pedestrian region is removed from the video.

7. The method according to claim 6, wherein: The step of generating pedestrian behavior prediction data by inputting grouped feature data into separate processing modules and cascading the output results of the corresponding processing modules comprises the following steps: Inputting the pedestrian posture feature data into a first recurrent neural network module; Acquire the feature encoding of the pedestrian posture feature data from the first recurrent neural network module; Inputting the feature encoding of the pedestrian posture feature data and the pedestrian bounding box information into a second recurrent neural network module; Acquire the feature encoding of the pedestrian posture feature data and the pedestrian bounding box from the second recurrent neural network module; Inputting the vehicle dynamic data, the feature encoding of the pedestrian posture feature data and the pedestrian bounding box into a third recurrent neural network module; and The feature encoding corresponding to the posture feature data of the pedestrian, the pedestrian bounding box and the vehicle dynamic data are obtained from the third recurrent neural network module.

8. The method according to claim 6, wherein: The step of generating pedestrian behavior prediction data by inputting grouped feature data into separate processing modules and cascading the output results of the corresponding processing modules comprises the following steps: Inputting the local context feature data, the global context feature data and the scene context feature data into CNN; Inputting the visual feature data of the local context feature data, the global context feature data and the scene context feature data from the CNN into the first recurrent neural network module to the third recurrent neural network module; and The feature encodings of the local context features, the global context features, and the scene context features output from the first recurrent neural network module to the third recurrent neural network module are cascaded and input into the fourth recurrent neural network module.

9. The method according to claim 6, wherein: The step of generating the pedestrian behavior prediction data by inputting the grouped feature data into separate processing modules and cascading the output results of the corresponding processing modules comprises the following steps: Inputting the pedestrian region obtained from the video input and the pedestrian surrounding region after the pedestrian region in the video is removed into a corresponding 3D convolutional network; By reducing the dimension of the feature map obtained from the corresponding 3D convolutional network to the corresponding maximum pooling layer according to the feature map; Convert each reduced feature map into a one-dimensional vector, and then pass the converted vector to the fully connected layer; and The result of cascading the input / output neurons is generated and passed to the neural network.

10. The method according to claim 6, wherein: The step of generating the pedestrian behavior prediction data by inputting the grouped feature data into separate processing modules and cascading the output results of the corresponding processing modules comprises the following steps: Inputting the feature encoding from the observation module into a first attention module; Inputting the feature encoding from the context module into a second attention module; Inputting the output result of the second attention module into a third attention module after being cascaded with the feature code from the convolution module; and The pedestrian behavior prediction data is generated by cascading the feature encodings from the first attention module and the third attention module, and then inputting the cascaded features into the fourth attention module.

11. The method according to claim 1, wherein: The step of assessing risk based on the trajectory prediction data, the behavior prediction data, and the surface classification result of the video input comprises the following steps: Generating a segmentation map by performing semantic segmentation on the video input; generating a ground area map by identifying ground areas using the segmentation map; and The pedestrian risk is assessed based on a pedestrian surrounding area in the video and the ground area map, wherein the pedestrian surrounding area is an area around a bounding box corresponding to the pedestrian in the video input.

12. The method according to claim 11, wherein: The step of generating a ground area map by identifying the ground area using the segmentation map comprises the following steps: generating a ground region mask, wherein pixels belonging to the ground region are set to true; calculating a cumulative probability of the ground region as a function of time by applying the ground region mask to the segmentation map; and The ground area map indicating to which ground area each pixel belongs is generated based on the cumulative probability.

13. The method according to claim 11, wherein: The step of assessing risk based on the trajectory prediction data, the behavior prediction data and the surface classification results in the video input includes the step of extracting the area around the pedestrian from the video and the bounding box, The step of extracting the area around the pedestrian in the video includes the following steps: Extracting a first feature map including visual features for the pedestrian's surroundings based on the video and the bounding box; and A second feature map including ground features for the pedestrian and the pedestrian's surroundings is extracted based on the video, the bounding box, and the ground area map.

14. The method according to claim 13, wherein: The step of extracting the first feature map comprises the following steps: generating a first cropped video from the video input by cropping a region including the bounding box corresponding to a pedestrian by a predetermined multiple of a size of the bounding box; removing an area corresponding to the bounding box from the first cropped video; and The first feature map including visual features of the pedestrian's surroundings is extracted by inputting the video from which the area corresponding to the bounding box is removed into a pre-trained visual feature extraction network.

15. The method according to claim 14, wherein: The step of extracting the second feature map comprises the following steps: generating a second cropped video by converting the ground area map into an RGB video and then cropping an area that is the same as that of the first cropped video; By inputting the second cropped video into a pre-trained road area feature extraction network, the second feature map including ground features for the pedestrian area and the area surrounding the pedestrian is extracted; and The probability-based pedestrian risk is output by cascading the first feature map and the second feature map.

16. The method according to claim 11, wherein: The step of assessing risk based on the trajectory prediction data, the behavior prediction data and the surface classification result in the video input also includes the step of assessing the semantic position data of the pedestrian based on the bounding box and the ground area map.

17. The method according to claim 16, wherein: The step of evaluating the semantic position data of the pedestrian based on the bounding box and the ground area map comprises the following steps: estimating a pedestrian foot region from the bounding box; and The category corresponding to the pixel on the ground area map that contacts the pedestrian's foot area the most is evaluated as the semantic position data of the pedestrian.

18. The method according to claim 17, wherein: The step of estimating the pedestrian foot region from the bounding box estimates a bottom region having a size that is a predetermined ratio of the size of the bounding box as the pedestrian foot region.

19. A system for predicting pedestrian safety information based on video, the system comprising: a communication module for receiving video input captured by a camera installed at a predetermined location; a memory for storing a program for assessing pedestrian risk based on the captured video input; as well as A processor, by running the program stored in the memory, is used to generate pedestrian trajectory prediction data based on the video input, to generate pedestrian behavior prediction data based on the video input, and to assess pedestrian safety risks based on the trajectory prediction data, the behavior prediction data and the surface classification results in the video input.

20. A system for predicting pedestrian safety information based on video, the system comprising: a communication module for receiving video input captured by a camera installed at a predetermined location; a memory for storing a program for assessing pedestrian risk based on the captured video input; as well as The processor, by running the program stored in the memory, is configured to: extracting visual feature data for future pedestrian trajectory prediction based on the video input, and generating pedestrian trajectory prediction data by inputting the visual feature data into a recurrent encoder and a decoder, extracting a plurality of visual input feature data and non-visual input feature data from the video input and from the vehicle dynamic sensor, and generating pedestrian behavior prediction data by grouping and inputting corresponding extracted input feature data into corresponding processing modules and then cascading output results, and Pedestrian semantic position data and the pedestrian safety risk are evaluated based on the trajectory prediction data, the behavior prediction data and the surface classification results in the video input.

Images