Personnel behavior supervision system and method based on vehicle-mounted video

Abstract

The invention discloses a personnel behavior supervision system and method based on a vehicle-mounted video, and the method comprises the steps: building a model switching buffer region in a region corresponding to each block data set, calibrating a switching point on a map, enabling a sub-model to be started when a vehicle passes through the calibrated switching point, and enabling a main model and the sub-model to run at the same time; judging whether the main model and the sub-model conform to a switching condition or not, switching to the sub-model if the main model and the sub-model conform to the switching condition, and quitting the operation of the sub-model and keeping the operation of the main model if the main model and the sub-model do not conform to the switching condition; and according to the driving route, loading the model used by the area to be passed in advance. A sub-model is additionally trained on the basis of the main model, and the sub-model is applied to a model training data acquisition area, so that pedestrian behavior prediction of the area is more accurate than that of the main model; the route which the vehicle is about to pass through can be loaded in advance according to the current situation direction of the vehicle, the corresponding model on the route is loaded in advance, and the model switching time is shortened.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a personnel behavior monitoring system and method based on vehicle-mounted video. Background Technology

[0002] In vehicle driving scenarios, safety is obviously a top priority. Safe driving not only protects the property safety of vehicles and other objects on the road, but also relates to the personal safety of the driver and other drivers or pedestrians on the road. Every traffic accident serves as a reminder to drivers of the importance of safe driving.

[0003] Therefore, based on considerations of safe driving, modern vehicles are generally equipped with safety warning technologies. For example, some existing new energy vehicles are equipped with onboard terminals for assisted driving and intelligent driving, aiming to reduce traffic accidents. Existing technologies use deep learning to identify (predict) pedestrian crossing intentions and predict pedestrian trajectories. By using onboard cameras to perceive the trajectory, actions, and postures of pedestrians on the roadside, and combining this with the dynamic information of the autonomous vehicle itself (such as speed and acceleration), the technology can identify the crossing intentions of specific pedestrians and predict their trajectories, making it easier for autonomous vehicles to avoid pedestrians and thus improve road safety. However, it still has the following shortcomings: although a large number of images are used for model training, the trained model cannot predict pedestrian behavior in specific areas, leading to misjudgments of pedestrian behavior in specific areas. Summary of the Invention

[0004] The purpose of this invention is to provide a personnel behavior monitoring system based on vehicle-mounted video to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a personnel behavior monitoring system based on vehicle video, comprising a CPU, a task sequence, a control module, a main model calculation module, and a sub-model calculation module; CPU, used for storing input images and configuring task sequences; The task sequence, connected to the CPU, is used to store task information for the main model and sub-models to be computed; The control module connects to the task sequence and is used to configure the main model operation module, the sub-model operation module, the read and write of the external storage, and the switching of tasks provided by the task sequence. External storage, connected to the CPU, is used to store images to be processed; Both the main model operation module and the sub-model operation module are connected to the control module, and perform model operations based on the trained main model and sub-model.

[0006] Preferably, the main model operation module and the sub-model operation module use the main model and the sub-model respectively for calculation, and perform convolutional neural network calculations based on the input image to predict pedestrian behavior.

[0007] Preferably, the main model calculation module is in a long-term running state, and the control module controls the continued operation of the main model calculation module, the operation of the sub-model calculation module, and the simultaneous operation of the main model calculation module and the sub-model calculation module according to the priority of the CPU adding tasks in the task sequence.

[0008] Preferably, the sub-model is based on the main model and is obtained by training using a block dataset.

[0009] Preferably, the block dataset includes image data acquired from areas with high pedestrian traffic and areas with frequent traffic accidents.

[0010] Preferably, the regions corresponding to the block datasets are marked on the map, so that when a vehicle travels to the marked region, it automatically switches to a sub-model trained on the specific data.

[0011] A method for monitoring personnel behavior based on in-vehicle video includes the following steps: 1) Establish a model switching buffer in the region corresponding to each block dataset. By marking switching points on the map, the sub-model starts when the vehicle passes the marked switching point, and the main model and sub-model run simultaneously. 2) Determine whether the main model and sub-model meet the switching conditions. If they meet the switching conditions, switch to the sub-model. If they do not meet the switching conditions, the sub-model exits the operation, and the main model continues to run. 3) Based on the driving route, preload the models used for the areas to be traversed.

[0012] Preferably, the model switching buffer is a region with few pedestrians before entering the region corresponding to the sub-model.

[0013] Preferably, the model switching condition in step 2) is as follows: the obtained pedestrian image is used as input, behavior prediction is performed by the main model and the sub-model respectively, and the prediction result is compared with the actual pedestrian position to determine whether the sub-model meets the switching condition. The model whose prediction result is closer to the actual pedestrian position is used as the switching model.

[0014] The technical effects and advantages of this invention are as follows: 1. By training a sub-model on the basis of the main model and applying the sub-model to the model training data collection area, the prediction of pedestrian behavior in the area is more accurate than that of the main model. 2. Mark the model on the map, and it will automatically switch when the vehicle is about to pass through the area, which is very convenient; 3. The route that the vehicle is about to travel can be loaded in advance based on the current situation and direction of the vehicle, and the corresponding model on the route can be loaded in advance to reduce the model switching time; 4. Set up a model switching buffer to complete model selection and switching within the buffer, avoiding unexpected situations that may be caused by missing pedestrian behavior predictions due to model switching gaps; 5. By controlling the operation of the main model operation module and the sub-model operation module through the control module, and by sorting the task priority of the task sequence, serial and parallel operation of multiple tasks can be realized, which improves the real-time performance and flexibility of the system. Attached Figure Description

[0015] Figure 1 This is a system module diagram of the present invention; Figure 2 This is a schematic diagram of the loading of a regional sub-model in the vehicle's driving route according to the present invention; Figure 3 A schematic diagram showing the main model switching to a sub-model via the model switching buffer; Figure 4 This is a flowchart of the method of the present invention. Detailed Implementation

[0016] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that these descriptions are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0017] This invention provides, for example Figure 1 The system shown is a vehicle-mounted video-based personnel behavior monitoring system, comprising a CPU, a task sequence, a control module, a main model computation module, and a sub-model computation module: the CPU is used for storing input images and configuring the task sequence; the task sequence is used to store task information for the main model and sub-models to be computed; the control module is used to configure the main model computation module, sub-model computation module, external memory read / write, state interaction between modules, and task switching provided by the task sequence; the external memory is used to store images to be processed; and the internal memory is used for data buffer input and output.

[0018] The main model computation module is in a long-term running state. The CPU stores the image task to be recognized in the external memory and adds a task to the task sequence through the APB (Advanced Peripheral Bus). The external memory and internal memory transfer data through AXI (Next Generation High Performance Bus). After detecting a task to be executed in the task sequence, the control module starts configuring the main model computation module, reads the input image from the external memory, performs convolutional neural network inference, and predicts pedestrian behavior. When the sub-model computation module needs to intervene in the computation, the task priority of the sub-model computation module is increased, enabling the sub-model computation module to predict pedestrian behavior. According to the task priority, the control module controls the computation of the main model computation module and the sub-model computation module, and determines the switching between the main model and the sub-model according to the needs.

[0019] The main model computation module and the sub-model computation module use the main model and the sub-model for computation, respectively. They perform convolutional neural network computation based on the input image to predict pedestrian behavior.

[0020] As an embodiment of the present invention, the main model and the sub-model are established using the following method: Main model: includes input module, feature extraction module, encoding module, decoding module, and output module. Input module: Used to acquire input data, namely pedestrian historical trajectory data, including existing datasets, such as the open-source public datasets ETH and UCY datasets. The pedestrian historical trajectory data is input into the YOLOv8 visual neural network model. The model parses the input historical trajectory information and processes various interactive feature information related to pedestrian movement. The above interactive feature information includes pedestrian coordinate information, scene interaction information, semantic information, speed information, and pose information, etc., and uses this information to represent the pedestrian's movement behavior and social interaction relationship.

[0021] Feature extraction module: Extracts various interaction feature information from the input data so that the model can better understand the trajectory data; Encoding module: Encodes interactive feature information into an intermediate state for use in the subsequent decoding stage when generating the trajectory.

[0022] Decoding module: It analyzes the intermediate states generated in the encoding stage and predicts the future trajectory of the pedestrian. This application uses an autoregressive model to generate the trajectory, that is, it uses the points generated in the previous time step as input and then generates the points in the next time step, thereby completing the generation of the pedestrian's movement trajectory.

[0023] Output module: Outputs the trajectory generated in the decoding stage as the final result. The output format is the coordinate points of the trajectory (including single trajectory and multiple trajectory). Evaluation metrics are used to evaluate the predictive performance of the model, including average displacement error and final displacement error.

[0024]

[0025] Where ADE refers to the average Euclidean distance difference between each predicted location and each ground truth location, FDE represents the average Euclidean distance difference between the predicted endpoint location and the ground truth endpoint location, and Nt represents the total number of pedestrians in the target trajectory at time t. Represents all predicted objects. It is the moment of observation. It is the predicted time. and Let _____ represent the true coordinates and the predicted coordinates, respectively, and let _____ be the norm. Represents the actual trajectory. This indicates the predicted trajectory.

[0026] Sub-models: Sub-models are based on the main model and trained using specific data, enabling them to achieve more accurate outputs when given specific types of data as input. Specifically, a map-based block dataset is constructed, marking points on the map where separate block datasets need to be created. These points correspond to the actual operating areas of the sub-models, such as areas with high pedestrian traffic or frequent traffic accidents. Thresholds can be set, such as based on pedestrian traffic or the number of traffic accidents, to determine whether to create a separate block dataset for a particular area. The specific requirements are determined based on actual usage needs. The block dataset includes not only pedestrian behavior images but also data on traffic signs and everyday objects such as flowers, trees, and shrubs that affect pedestrian movement trajectories, making the data more diverse and facilitating more accurate judgment of human behavior. The model is trained using each block dataset as input to obtain a sub-model that corresponds to the region of each block dataset.

[0027] As an embodiment of the present invention, model switching requires time. To solve this problem, the model to be used is preloaded to save time. Specifically, according to the driving route, sub-models of areas that may be reached are preloaded. This can be achieved by adding the sub-models of the upcoming areas to the task sequence and setting their priority according to the order in which they are passed. Figure 2As shown, the vehicle travels along the route and will first enter the first intersection, namely sub-model operating area A1. After passing through this area, there will be three driving routes, which pass through sub-model operating areas B2, B2, and B3 respectively. The sub-models corresponding to sub-model operating areas B1, B2, and B3 will be selected as the models to be used. When the vehicle moves to sub-model operating area B3, after passing through this area, there will be three driving routes, which pass through sub-model operating areas C1, C2, and C3 respectively. The sub-models corresponding to sub-model operating areas C1, C2, and C3 will be selected as the models to be used. When the vehicle moves to sub-model operating area C3, there will be two driving routes... The driving route corresponds to the sub-model operating areas D1 and D2, until finally reaching D2. Images of the driving process are input into the sub-model through video tools such as cameras. The sub-model is a model specifically trained for the current area, which can predict pedestrian behavior in the current area more accurately, facilitating assisted driving and autonomous driving. This function is more suitable for use with navigation devices. When users use navigation devices, they can know the user's driving route. For example, if the navigation is from point O to point D2, it can be known that the vehicle will pass through the sub-model operating areas A1, B3, and C3. The sub-models A1, B3, and C3 on the driving route can be loaded accurately in advance. To reduce the system load, only one sub-model that is about to arrive needs to be loaded in advance.

[0028] As an embodiment of the present invention, such as Figure 3 As shown, there is a gap between the main model and the sub-model. To adapt to model switching under complex road conditions and reduce the potential for unexpected pedestrian behavior prediction gaps caused by the model switching gap, a model switching buffer is established in the region corresponding to each block of the dataset. The model switching buffer is the area with fewer pedestrians before entering the region corresponding to the sub-model. Model switching is achieved by marking switching points on the map. Before switching, the main model is in a long-term running state. When a vehicle enters the marked switching point, the sub-model starts. At this time, the main model and the sub-model run simultaneously, using the acquired pedestrian images as input, and performing behavior prediction through the main model and the sub-model respectively. The prediction results are compared with the actual pedestrian positions to determine if the sub-model meets the switching conditions. If the prediction results of both the main model and the sub-model are the same as the actual pedestrian positions, the sub-model is deemed to meet the switching conditions, and the main model stops running to reduce system usage, lower load, and improve operating efficiency. When a vehicle leaves the area corresponding to the sub-model, it passes through the model buffer and switches back to the main model. If the prediction result of the main model is the same as the actual pedestrian position but different from the prediction result of the sub-model, the sub-model stops running. When a vehicle passes through the area corresponding to the sub-model, the main model continues to run to predict pedestrian behavior.

[0029] As an embodiment of the present invention, in order to improve the accuracy of long-distance small-scale target detection during the main model building process, a small-scale detection layer is added to the YOLOv8 network framework to improve the detection capability of small targets; in view of the characteristics of small-scale pedestrian targets, a set of preset small-scale anchor boxes is added to make it more suitable for the added small-scale detection layer.

[0030] like Figure 4 As shown in the figure, as an embodiment of the present invention, the present invention also provides a method for monitoring personnel behavior based on vehicle-mounted video, comprising the following steps: Step 1: Establish a model switching buffer in the region corresponding to each block dataset. The model switching buffer is the region with fewer pedestrians before entering the region corresponding to the sub-model. By marking the switching point on the map, the sub-model starts when the vehicle passes the marked switching point, and the main model and sub-model run simultaneously. Step 2: Determine whether the main model and sub-model meet the switching conditions. Using the acquired pedestrian image as input, perform behavior prediction through the main model and sub-model respectively, and compare the prediction results with the actual pedestrian positions to determine whether the sub-model meets the switching conditions. The model whose prediction results are closer to the actual pedestrian positions is used as the switching model. If the switching conditions are met, switch to the sub-model. If the switching conditions are not met, the sub-model exits the operation, and the main model continues to run. Step 3: Based on the driving route, preload the models used for the areas to be traversed.

[0031] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A personnel behavior monitoring system based on vehicle-mounted video, characterized in that: It includes CPU, task sequence, control module, main model operation module and sub-model operation module; CPU, used for storage of input images and configuration of task sequences; Task sequence, connected to the CPU, used to store the task information of the main model and sub-model to be calculated; The control module is connected to the task sequence and is used to configure the main model operation module, the sub-model operation module, the reading and writing of the external memory and the switching of the tasks provided by the task sequence; An external memory, connected to the CPU, for storing images to be processed; The main model operation module and the sub-model operation module are both connected to the control module to perform model operations according to the trained main model and sub-model.

2. According to claim 1, a personnel behavior monitoring system based on vehicle-mounted video is characterized in that: The main model operation module and the sub-model operation module respectively use the main model and the sub-model to perform calculations, perform convolutional neural network calculations based on the input image, and predict pedestrian behavior.

3. The personnel behavior monitoring system based on vehicle-mounted video according to claim 1 is characterized in that: The main model operation module is in a long-term running state, and the control module controls the continued operation of the main model operation module, the operation of the sub-model operation module, and the simultaneous operation of the main model operation module and the sub-model operation module according to the priority of the task added by the CPU in the task sequence.

4. The personnel behavior monitoring system based on vehicle-mounted video according to claim 1 is characterized in that: The sub-model is based on the main model and is obtained by training with the block data set.

5. The personnel behavior monitoring system based on vehicle-mounted video according to claim 4 is characterized in that: The block data set includes image data obtained from areas with large pedestrian flow and areas with frequent traffic accidents.

6. A personnel behavior monitoring system based on vehicle-mounted video according to claim 5, characterized in that: The area corresponding to the block data set is marked on the map, so that when a vehicle drives to the marked area, it automatically switches to the sub-model trained with specific data.

7. The method for monitoring personnel behavior based on vehicle-mounted video according to claim 1, characterized in that: The steps include: 1) Establish a model switching buffer in the area corresponding to each block data set. By marking the switching point on the map, the sub-model is started when the vehicle passes the marked switching point, and the main model and sub-model run simultaneously; 2) Determine whether the main model and the sub-model meet the switching conditions. If they meet the switching conditions, switch to the sub-model. If they do not meet the switching conditions, the sub-model exits and keeps the main model running. 3) According to the driving route, load the model used in the area to be passed in advance.

8. The method for monitoring personnel behavior based on vehicle-mounted video according to claim 7 is characterized in that: The model switching buffer is an area with few pedestrians before entering the area corresponding to the sub-model.

9. The method for monitoring personnel behavior based on vehicle-mounted video according to claim 7, characterized in that: The model switching condition in step 2) is: using the obtained pedestrian image as input, performing behavior prediction through the main model and the sub-model respectively, and comparing the prediction result with the actual pedestrian position to determine whether the sub-model meets the switching condition, and using the model whose prediction result is closer to the actual pedestrian position as the switching model.

Images