A rescue vehicle empty or heavy load state identification system and method based on multi-source data

Abstract

The application discloses a kind of based on multi-source data's rescue vehicle empty heavy load state identification system and method, it is related to rescue vehicle state identification technical field, comprising: the weight induction during the history of rescue vehicle execution task period and monitoring video data are collected, the history video of rescue vehicle loading is screened out, confirms historical loading time interval, the historical weight data sensed in loading process is arranged and analyzed, generates historical sensing data broken line chart, filters out abnormal broken line chart, sets up traction abnormality judgment interval and sensor abnormality judgment interval, obtains current task execution abnormality maintenance duration, compares current task execution abnormality maintenance duration with traction abnormality judgment interval and sensor abnormality judgment interval respectively, selectively eliminates image frame, after the elimination processing selection is completed, the empty heavy load state identification of rescue vehicle and work event identification are carried out, pre-eliminate the image frame that possibly causes the error of identification result, guarantee the accuracy of identification result.

Description

Technical Field

[0001] This invention relates to the field of rescue vehicle status recognition technology, specifically a rescue vehicle empty / heavy status recognition system and method based on multi-source data. Background Technology

[0002] Roadside assistance vehicles perform transport tasks for vehicles that have broken down on the road. By installing high-definition cameras on the vehicle's frame, they can capture real-time video footage on the vehicle's tablet platform. The real-time video footage is then wirelessly transmitted to a smart terminal. The smart terminal performs AI recognition on the captured video and reports the recognition results to the roadside assistance monitoring platform. The platform can then collect statistics on the time and location of the roadside assistance vehicle's transport of the broken-down vehicle, the vehicle's empty or loaded status, and information on the vehicle's loading and carrying of cargo, facilitating intelligent management of roadside assistance work.

[0003] However, during the loading process of a rescue vehicle, a breakdown may occur, causing the vehicle to roll backward. If the video of this backward movement is input into an AI model to identify the rescue vehicle's empty or loaded status, as well as loading and unloading events, for example, by inputting it into an improved YoloV5 algorithm model, the backward movement might be mistakenly identified as the unloading process. This reduces the accuracy of the identification results for the rescue vehicle's loading and unloading operations, hindering the rescue monitoring platform from receiving accurate rescue information and enabling intelligent management of road rescue work. Summary of the Invention

[0004] The purpose of this invention is to provide a rescue vehicle empty / heavy load status identification system and method based on multi-source data, so as to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a rescue vehicle empty / heavy load status recognition system based on multi-source data, the system comprising a sensor information acquisition module, a sensor data mapping and analysis module, an image frame elimination and selection module, and a status recognition module;

[0006] The sensor information acquisition module collects weight sensing data and monitoring video data from the rescue vehicle's historical missions.

[0007] The sensor data mapping and analysis module filters out historical videos of the rescue vehicle loading, confirms the historical loading time interval, arranges and analyzes the historical weight data sensed during the loading process, generates a historical sensor data line chart, filters out abnormal line charts through matrix analysis, and sets traction abnormal judgment intervals and sensor abnormal judgment intervals.

[0008] The image frame elimination selection module obtains the duration of the current task execution anomaly, compares the duration of the current task execution anomaly with the traction anomaly judgment interval and the sensor anomaly judgment interval respectively, and performs selective elimination processing on the image frames of the monitored video based on the comparison results.

[0009] After the rejection process is completed, the status recognition module identifies the empty and heavy load status of the rescue vehicle and the operation event.

[0010] Preferably, the sensing information acquisition module includes: a weight sensing data acquisition unit and a historical monitoring data acquisition unit;

[0011] The weight sensing data acquisition unit is used to collect historical weight data sensed by the rescue vehicle during previous missions. The rescue vehicle is equipped with a weight sensor, which is used to collect weight data when the rescue vehicle is loading and unloading cargo.

[0012] The historical monitoring data acquisition unit is used to collect historical video data captured by the cameras installed on the rescue vehicle.

[0013] Preferably, the sensing data mapping and analysis module includes: a sensing data arrangement unit, a line chart generation and analysis unit, and an anomaly judgment interval setting unit;

[0014] The sensing data arrangement unit is used to filter out video data of the rescue vehicle loading from historical video data, and to confirm the start time and end time of the filtered video. The start time refers to the start time of loading when the rescue vehicle randomly tows the disabled vehicle, and the end time refers to the end time of loading. When loading, the rescue vehicle needs to tow the disabled vehicle to the flat platform of the rescue vehicle. The unit filters out the historical weight data sensed from the start time to the end time, and combines the filtered historical weight data and sensing time data into data points and arranges them in a rectangular coordinate system. The horizontal axis of the rectangular coordinate system represents the weight data sensing time, and the vertical axis represents the historical weight data. The weight data sensing time is distributed at equal intervals, that is, the historical weight data collection time interval is equal.

[0015] The line chart generation and analysis unit is used to connect adjacent data points and generate several historical sensing data line charts. Each line chart is analyzed: it is determined whether the slope between adjacent data points is negative. If it is, the change feature of the adjacent data points is recorded as feature value 0; otherwise, it is recorded as feature value 1. After the analysis of a line chart is completed, a feature column matrix is ​​generated based on the statistical feature values. A benchmark judgment column matrix is ​​set. Based on the feature column matrix and the benchmark judgment column matrix, it is determined whether the corresponding line chart is an abnormal line chart, and all abnormal line charts are filtered out.

[0016] The anomaly judgment interval setting unit is used to obtain the abnormal time interval in each abnormal line graph. The abnormal time interval is obtained as follows: obtain the abnormal points in each abnormal line graph, identify the abnormal point with the smallest horizontal coordinate value and the abnormal point with the largest horizontal coordinate value. The horizontal coordinate of the abnormal point with the smallest horizontal coordinate value is the lower limit of the abnormal time interval, and the horizontal coordinate of the abnormal point with the largest horizontal coordinate value is the upper limit of the abnormal time interval. Retrieve the video monitored within the abnormal time interval and determine whether the faulty vehicle being towed by the rescue vehicle during loading in the video exhibits a backward movement. Based on the judgment result, the abnormal line graph is divided into a first abnormal line graph and a second abnormal line graph. The first abnormal line graph represents the towing abnormal data. The first abnormal line chart is a line chart of sensor abnormal data. Abnormal points in the first abnormal line chart are obtained, and the difference in horizontal coordinates between the abnormal point with the smallest and largest horizontal coordinates is identified. This difference is used as the duration of traction abnormality, resulting in several traction abnormality durations. Similarly, abnormal points in the second abnormal line chart are obtained, and the difference in horizontal coordinates between the abnormal point with the smallest and largest horizontal coordinates is identified. This difference is used as the duration of sensor abnormality, resulting in several sensor abnormality durations. Based on these durations and the sensor abnormality durations, traction abnormality judgment intervals and sensor abnormality judgment intervals are set.

[0017] Preferably, the image frame elimination selection module includes: a current data analysis unit and an image frame elimination selection unit;

[0018] The current data analysis unit is used to collect and analyze the weight data sensed by the weight sensor when the rescue vehicle is actually performing a task. If the weight data changes from decreasing to increasing, it is determined that the current task is abnormal. The unit also counts the time when the weight data starts to decrease and the time when it starts to increase after decreasing, and obtains the duration of the current task execution abnormality.

[0019] The image frame rejection selection unit is used to compare the duration of the current task execution anomaly with the traction anomaly judgment interval and the sensor anomaly judgment interval, respectively. If the duration of the current task execution anomaly is within the traction anomaly judgment interval, the cause of the current task execution anomaly is determined to be the backward movement of the malfunctioning vehicle due to traction anomaly during loading. If the duration of the current task execution anomaly is within the sensor anomaly judgment interval, the cause of the current task execution anomaly is determined to be the sensor malfunction of the rescue vehicle. If the cause of the current task execution anomaly is the backward movement of the malfunctioning vehicle due to traction anomaly during loading, the image frames of the video monitored within the current task execution anomaly time interval will be rejected. Otherwise, no image frame rejection processing will be performed.

[0020] Preferably, the status recognition module includes: a video stream transmission unit, an idle / overload status recognition unit, and a job event analysis unit;

[0021] The video stream transmission unit is used to transmit the monitored video to the vehicle edge device after the elimination process is completed. The vehicle edge device is used to preprocess each frame of the video. The preprocessing is to perform enhancement, noise reduction and other processing on the image to improve the image quality and improve the accuracy of state recognition.

[0022] The empty / heavy load state identification unit is used to calibrate the flatbed platform area of ​​the rescue vehicle. It uses an improved YOLOv5 algorithm to identify whether there are faulty vehicles in the calibrated area. It sets a detection time, counts the number of image frames containing faulty vehicles and the number of image frames without faulty vehicles identified within the detection time, and sets a frame count threshold. If the number of image frames containing faulty vehicles exceeds the frame count threshold, it is determined that the rescue vehicle is in a heavy load state; if the number of image frames without faulty vehicles exceeds the frame count threshold, it is determined that the rescue vehicle is in an empty load state.

[0023] The operation event analysis unit is used to identify rescue vehicle operation events: if the rescue vehicle is identified as empty in the previous detection period and as heavily loaded in the next detection period, it is determined that the rescue vehicle is loading; if the rescue vehicle is identified as heavily loaded in the previous detection period and as empty in the next detection period, it is determined that the rescue vehicle is unloading.

[0024] A method for identifying the empty and heavy load status of rescue vehicles based on multi-source data includes the following steps:

[0025] S1: Collect weight sensor data and monitoring video data from the rescue vehicle's historical missions;

[0026] S2: Filter out historical videos of the rescue vehicle loading, confirm the historical loading time interval, arrange and analyze the historical weight data sensed during the loading process, generate a line chart of historical sensing data, filter out abnormal line charts through matrix analysis, and set the traction anomaly judgment interval and sensor anomaly judgment interval.

[0027] S3: Obtain the duration of the current task execution anomaly, compare the duration of the current task execution anomaly with the traction anomaly judgment interval and the sensor anomaly judgment interval respectively, and selectively remove the image frames of the monitored video based on the comparison results.

[0028] S4: After the elimination process is completed, the empty and heavy load status of the rescue vehicle and the operation event are identified.

[0029] Preferably, in step S2: historical videos of the rescue vehicle loading are selected from the monitoring video data during a random mission performed by the rescue vehicle, with the start time point T1 and the end time point T2 of the selected videos. The time interval of a random historical loading is confirmed as [T1, T2]. Historical weight data sensed within the time interval [T1, T2] are selected to obtain the historical weight set M = {M1, M2, ... M}. k The time it takes for the weight sensor to detect the corresponding weight is t = {t1, t2, ..., t}. k Within the time interval [T1, T2], a total of k data points were sensed. The sensing times of the weight data were evenly distributed, meaning that the historical weight data collection time intervals were equal, for example: t2-t1=t3-t2, which were combined into data points {(t1, M1), (t2, M2), ..., (t... k M k Arrange the data points in a rectangular coordinate system, connect adjacent data points to generate a line graph of historical sensing data, and calculate the slope between the first group of adjacent data points as F, F=(M2-M1) / (t2-t1). If F is negative, record the change characteristic of the first group of adjacent data points as feature value 0; if F is not negative, record it as feature value 1. Determine whether the slope between all adjacent data points is negative, and generate a feature value set {A1,A2,...,A1}. k-1 The generated feature column matrix is: Set a benchmark column matrix Given that all elements in the baseline judgment column matrix are 1, and there are a total of k-1 elements, determine whether the corresponding line chart is an abnormal line chart according to the following method:

[0030] ;

[0031] in, This is the anomaly detection matrix for the line chart. If the matrix is ​​all zeros, then the corresponding line chart is determined not to be an abnormal line chart; otherwise, the corresponding line chart is determined to be an abnormal line chart. From the monitoring video data during the n missions of the rescue vehicle, n historical videos of the rescue vehicle loading are selected, n historical sensing data line charts are generated, and all abnormal line charts are selected.

[0032] Image recognition technology, combining computer vision and deep learning algorithms, can analyze the loading status of road rescue vehicles from multiple dimensions. Compared to relying on a single weight data source using weight sensors, it is less susceptible to environmental interference and can improve the reliability of empty / loaded identification results. However, considering that the YoloV5 algorithm requires recognition of each frame of the image, if a traction problem occurs during the loading process of the rescue vehicle, causing the disabled vehicle to back up, if the status of the image of the disabled vehicle backing up is identified, the operation event analysis may identify the rescue vehicle's status as heavily loaded in the previous detection period and empty in the next detection period. That is, the traction abnormality during loading is identified as the rescue vehicle unloading, which reduces the accuracy of the identification results of the rescue vehicle loading and unloading operation events. This invention integrates weight sensing technology and image recognition technology. Before identifying the status of the rescue vehicle, image frames of video that may show traction abnormalities during loading are pre-processed and then the empty / loaded status and operation events of the rescue vehicle are identified. By pre-eliminating image frames that may cause errors in the identification results, the accuracy of the identification results is ensured, which is conducive to the rescue supervision platform receiving accurate rescue work information and thus enabling intelligent management of road rescue work.

[0033] Preferably, the abnormal time intervals in each abnormal line graph are obtained, and the video monitored within the abnormal time interval is retrieved. It is determined whether the disabled vehicle being towed by the rescue vehicle during loading exhibits backward movement. Using image analysis technology, the positional changes of the disabled vehicle relative to a fixed reference point are continuously tracked to determine if the disabled vehicle's movement direction shows backward movement. If backward movement is observed, the abnormal line graph is classified as the first abnormal line graph; otherwise, it is classified as the second abnormal line graph. The first abnormal line graph is a traction abnormal data line graph, and the second abnormal line graph is a sensor abnormal data line graph. A total of m first abnormal line graphs are counted. Abnormal points in the first abnormal line graphs are obtained. If the slope between adjacent data points is negative, the corresponding two adjacent data points are considered abnormal points. The difference in the horizontal coordinate between the abnormal point with the smallest horizontal coordinate and the abnormal point with the largest horizontal coordinate is identified and used as the traction abnormality duration. The m traction abnormality durations are obtained as W = {W1, W2, ..., W...} m}, a total of r second-anomaly line graphs were counted, and the duration of the r sensor anomalies was obtained as w={w1,w2,...,w r}, we obtain the value that appears most frequently and is the smallest in set W as a, and the value that appears most frequently and is the largest as b, and set the traction anomaly judgment interval as [a, b]; we obtain the value that appears most frequently and is the smallest in set w as c, and the value that appears most frequently and is the largest as d, and set the sensor anomaly judgment interval as [c, d];

[0034] Considering that weight data anomalies are not always caused by traction anomalies, but may also be caused by malfunctions in the weight sensor itself, and that weight data anomalies caused by traction anomalies are sudden and generally short-lived, while weight data anomalies caused by weight sensor malfunctions are generally longer-lasting, by analyzing the patterns of data anomaly changes in the two situations through historical data analysis, traction anomaly judgment intervals and sensor anomaly judgment intervals are set. By comparing and analyzing the duration of data anomalies in actual tasks with the two intervals, it is helpful to reduce the probability of mistakenly rejecting image frames due to sensor malfunctions, and further improve the accuracy of the work event recognition results.

[0035] Preferably, in step S3: when the rescue vehicle is actually performing the task, the weight data sensed by the weight sensor is collected and analyzed. If the weight data changes from decreasing to increasing, it is determined that the current task execution is abnormal. The time point when the weight starts to decrease is V1, and the time point when the weight starts to increase after decreasing is V2. The duration of the current task execution abnormality is V2-V1. If V2-V1 is within the interval [a,b], it is determined that the cause of the current task execution abnormality is that the vehicle malfunctions and reverses due to traction abnormality during loading. The image frames of the video monitored during the time period from V1 to V2 are removed. If V2-V1 is within the interval [c,d], it is determined that the cause of the current task execution abnormality is that the sensor of the rescue vehicle is faulty. No image frame removal is performed. If V2-V1 is not within these two intervals, further investigation is conducted to determine the cause of the rescue vehicle's task execution abnormality.

[0036] Preferably, in step S4: after the elimination process is completed, the monitored video is transmitted to the vehicle-mounted edge device. The vehicle-mounted edge device preprocesses each frame of the video, calibrates the flat platform area of ​​the rescue vehicle, and uses the improved YOLOv5 algorithm to identify whether there are faulty vehicles in the calibrated area. A detection duration is set, and the number of image frames containing faulty vehicles and the number of image frames without faulty vehicles identified within the detection duration are counted. A frame count threshold is set. If the number of image frames containing faulty vehicles exceeds the frame count threshold, it is determined that the rescue vehicle is in a heavily loaded state; if the number of image frames without faulty vehicles exceeds the frame count threshold, it is determined that the rescue vehicle is in an unloaded state. If the rescue vehicle is identified as unloaded in the previous detection duration and as heavily loaded in the next detection duration, it is determined that the rescue vehicle is loading cargo; if the rescue vehicle is identified as heavily loaded in the previous detection duration and as unloaded in the next detection duration, it is determined that the rescue vehicle is unloading cargo.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] This invention integrates weight sensing technology and image recognition technology. Before identifying the status of the rescue vehicle, image frames from videos that may show abnormal traction during loading are pre-processed and then the empty and heavy load status of the rescue vehicle and operational events are identified. By pre-selecting image frames that may cause errors in the identification results, the accuracy of the identification results is ensured. This is beneficial for the rescue supervision platform to receive accurate rescue work information and thus carry out intelligent management of road rescue work.

[0039] Considering that weight data anomalies are not always caused by traction anomalies, but may also be caused by malfunctions in the weight sensor itself, and that weight data anomalies caused by traction anomalies are sudden and generally short-lived, while weight data anomalies caused by weight sensor malfunctions are generally longer-lasting, by analyzing the patterns of data anomaly changes in the two situations through historical data analysis, traction anomaly judgment intervals and sensor anomaly judgment intervals are set. By comparing and analyzing the duration of data anomalies in actual tasks with the two intervals, it is helpful to reduce the probability of mistakenly rejecting image frames due to sensor malfunctions, and further improve the accuracy of the work event recognition results. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the structure of a rescue vehicle empty / heavy load status recognition system based on multi-source data according to the present invention;

[0041] Figure 2 This is a flowchart illustrating a method for identifying the empty / heavy load status of a rescue vehicle based on multi-source data, according to the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Example 1: As Figure 1As shown, this embodiment provides a rescue vehicle empty / loaded status recognition system based on multi-source data. The system includes: a sensor information acquisition module, a sensor data mapping and analysis module, an image frame elimination and selection module, and a status recognition module. The sensor information acquisition module collects weight sensing data and monitoring video data from the rescue vehicle's historical missions. The sensor data mapping and analysis module filters historical videos of the rescue vehicle loading, confirms the historical loading time interval, arranges and analyzes the historical weight data sensed during loading, generates a historical sensor data line chart, and filters out abnormal line charts through matrix analysis, setting traction anomaly judgment intervals and sensor anomaly judgment intervals. The image frame elimination and selection module obtains the duration of the current mission execution anomaly, compares the duration of the current mission execution anomaly with the traction anomaly judgment interval and the sensor anomaly judgment interval respectively, and selectively eliminates image frames from the monitored video based on the comparison results. After the elimination and selection process is completed, the status recognition module performs empty / loaded status recognition and operational event recognition of the rescue vehicle.

[0044] The sensing information acquisition module includes a weight sensing data acquisition unit and a historical monitoring data acquisition unit. The weight sensing data acquisition unit is used to collect historical weight data sensed by the rescue vehicle during previous missions. The rescue vehicle is equipped with weight sensors, which are used to collect weight data when the rescue vehicle is loading and unloading cargo. The historical monitoring data acquisition unit is used to collect historical video data captured by cameras installed on the rescue vehicle.

[0045] The sensor data mapping and analysis module includes: a sensor data arrangement unit, a line chart generation and analysis unit, and an anomaly judgment interval setting unit. The sensor data arrangement unit is used to filter out video data of the rescue vehicle loading from historical video data, confirming the start and end times of the selected videos. The start time refers to the start time of loading when the rescue vehicle randomly tows the disabled vehicle, and the end time refers to the end time of loading. During loading, the rescue vehicle needs to tow the disabled vehicle onto its flatbed platform. The unit filters out historical weight data sensed from the start time to the end time, and combines the selected historical weight data with the sensing time data into data points arranged in a Cartesian coordinate system. The horizontal axis of the angular coordinate system represents the weight data sensing time, and the vertical axis represents historical weight data. The weight data sensing times are evenly distributed, meaning the historical weight data collection time intervals are equal. The line chart generation and analysis unit connects adjacent data points to generate several historical sensing data line charts. Each line chart is analyzed: it determines whether the slope between adjacent data points is negative. If so, the change characteristic of the adjacent data points is recorded as a feature value of 0; otherwise, it is recorded as a feature value of 1. After analyzing a line chart, a feature column matrix is ​​generated based on the statistical feature values. A benchmark judgment column matrix is ​​set, and the corresponding line chart is judged to be abnormal based on the feature column matrix and the benchmark judgment column matrix. Line charts are used to filter out all abnormal line charts. The abnormal judgment interval setting unit is used to obtain the abnormal time interval in each abnormal line chart. The abnormal time interval is obtained by: acquiring the abnormal points in each abnormal line chart, identifying the abnormal point with the smallest and largest x-coordinate values, setting the x-coordinate of the abnormal point with the smallest x-coordinate value as the lower limit of the abnormal time interval, and the x-coordinate of the abnormal point with the largest x-coordinate value as the upper limit of the abnormal time interval; retrieving the video monitored within the abnormal time interval, and determining whether the disabled vehicle being towed by the rescue vehicle during loading exhibits backward movement. Based on the judgment result, the abnormal line chart is divided into a first abnormal line chart and a second abnormal line chart. The first abnormal line chart... The first abnormal line chart is a traction anomaly data line chart, and the second abnormal line chart is a sensor anomaly data line chart. Abnormal points in the first abnormal line chart are obtained, and the difference in the horizontal coordinate between the abnormal point with the smallest and largest horizontal coordinate is identified. This difference is used as the duration of the traction anomaly, resulting in several traction anomaly durations. Similarly, abnormal points in the second abnormal line chart are obtained, and the difference in the horizontal coordinate between the abnormal point with the smallest and largest horizontal coordinate is identified. This difference is used as the duration of the sensor anomaly, resulting in several sensor anomaly durations. Based on these traction anomaly durations and sensor anomaly durations, traction anomaly judgment intervals and sensor anomaly judgment intervals are set.

[0046] The image frame elimination selection module includes a current data analysis unit and an image frame elimination selection unit. The current data analysis unit collects and analyzes weight data sensed by the weight sensor when the rescue vehicle is actually performing a task. If the weight data changes from a decrease to an increase, it determines that the current task execution is abnormal. It then calculates the time points when the weight data begins to decrease and when it begins to increase, obtaining the duration of the current task execution abnormality. The image frame elimination selection unit compares the duration of the current task execution abnormality with the traction abnormality judgment interval and the sensor abnormality judgment interval, respectively. If the duration of the current task execution abnormality is within the traction abnormality judgment interval, the cause of the current task execution abnormality is determined to be a backward movement of the malfunctioning vehicle due to traction abnormality during loading. If the duration of the current task execution abnormality is within the sensor abnormality judgment interval, the cause of the current task execution abnormality is determined to be a sensor malfunction of the rescue vehicle. If the cause of the current task execution abnormality is a backward movement of the malfunctioning vehicle due to traction abnormality during loading, image frames from the video monitored within the current task execution abnormality time interval will be eliminated. Otherwise, no image frame elimination processing will be performed.

[0047] The status recognition module includes: a video stream transmission unit, an empty / heavy load status recognition unit, and a work event analysis unit. The video stream transmission unit transmits the monitored video to the vehicle-mounted edge device after the removal process is completed. The edge device preprocesses each frame of the video, performing enhancement and noise reduction to improve image quality and status recognition accuracy. The empty / heavy load status recognition unit calibrates the rescue vehicle's flatbed platform area, using an improved YOLOv5 algorithm to identify whether there are faulty vehicles within the calibrated area. It sets a detection time and counts the number of image frames containing faulty vehicles identified within the detection time. The image frame count of vehicles without faults is set with a frame count threshold. If the number of image frames containing vehicles with faults exceeds the frame count threshold, the rescue vehicle is determined to be in a heavily loaded state; if the number of image frames of vehicles without faults exceeds the frame count threshold, the rescue vehicle is determined to be in an unloaded state. The operation event analysis unit is used to identify operation events of the rescue vehicle: if the rescue vehicle's state was identified as unloaded in the previous detection period and as heavily loaded in the next detection period, the rescue vehicle is determined to be loading cargo; if the rescue vehicle's state was identified as heavily loaded in the previous detection period and as unloaded in the next detection period, the rescue vehicle is determined to be unloading cargo.

[0048] Example 2: As Figure 2 As shown, this embodiment provides a method for identifying the empty and heavy load status of a rescue vehicle based on multi-source data. It is implemented using the identification system described in this embodiment and specifically includes the following steps:

[0049] S1: Collect weight sensor data and monitoring video data from the rescue vehicle's historical missions;

[0050] S2: Filter historical videos of the rescue vehicle loading, confirm the historical loading time interval, analyze the historical weight data sensed during the loading process, generate a line chart of historical sensing data, and filter out abnormal line charts using matrix analysis. Set traction anomaly judgment intervals and sensor anomaly judgment intervals: Filter historical videos of the rescue vehicle loading from the monitoring video data during a random mission of the rescue vehicle, obtaining the start time point T1 and end time point T2 of the selected videos, confirming the time interval of a random historical loading as [T1, T2]. Filter the historical weight data sensed within the time interval [T1, T2], obtaining the historical weight set as M = {M1, M2, ... M...} k The time it takes for the weight sensor to detect the corresponding weight is t = {t1, t2, ..., t}. k Within the time interval [T1, T2], a total of k data points were sensed. The sensing times for the weight data were evenly distributed, meaning that the historical weight data collection time intervals were equal, for example: t2-t1=t3-t2. These data points were combined to form data points {(t1, M1), (t2, M2), ..., (t... k M k Arrange the data points in a rectangular coordinate system, connect adjacent data points to generate a line graph of historical sensing data, and calculate the slope between the first group of adjacent data points as F, F=(M2-M1) / (t2-t1). If F is negative, record the change characteristic of the first group of adjacent data points as feature value 0; if F is not negative, record it as feature value 1. Determine whether the slope between all adjacent data points is negative, and generate a feature value set {A1,A2,...,A1}. k-1 The generated feature column matrix is: Set a benchmark column matrix Given that all elements in the baseline judgment column matrix are 1, and there are a total of k-1 elements, determine whether the corresponding line chart is an abnormal line chart according to the following method:

[0051] ;

[0052] in, This is the anomaly detection matrix for the line chart. If the matrix is ​​all zeros, then the corresponding line chart is determined not to be an abnormal line chart; otherwise, the corresponding line chart is determined to be an abnormal line chart. From the monitoring video data during the n missions of the rescue vehicle, n historical videos of the rescue vehicle loading are selected, n historical sensing data line charts are generated, and all abnormal line charts are selected.

[0053] For example: if the eigenvalue set is {A1,A2,A3,A4}={1,1,0,1}, the generated feature column matrix is... Set a benchmark column matrix The anomaly detection matrix of the line chart is obtained. If the anomaly detection matrix of a line chart is not a matrix of all zeros, then the corresponding line chart is determined to be an abnormal line chart.

[0054] Obtain the abnormal time interval from each abnormal line graph. Retrieve the monitored video within the abnormal time interval and determine if the disabled vehicle being towed by the rescue vehicle during loading exhibits any backward movement. If backward movement is observed, classify the abnormal line graph as the first abnormal line graph; otherwise, classify it as the second abnormal line graph. The first abnormal line graph is a traction abnormal data line graph, and the second abnormal line graph is a sensor abnormal data line graph. A total of m first abnormal line graphs are identified. Abnormal points within these first abnormal line graphs are obtained. If the slope between adjacent data points is negative, the corresponding two adjacent data points are considered abnormal points. The difference in horizontal coordinate between the abnormal point with the smallest and largest horizontal coordinate is used as the duration of the traction abnormality. The durations of the m traction abnormalities are then obtained as W = {W1, W2, ..., W...}. m}, a total of r second-anomaly line graphs were counted, and the duration of the r sensor anomalies was obtained as w={w1,w2,...,w r}, we obtain the value that appears most frequently and is the smallest in set W as a, and the value that appears most frequently and is the largest as b, and set the traction anomaly judgment interval as [a, b]; we obtain the value that appears most frequently and is the smallest in set w as c, and the value that appears most frequently and is the largest as d, and set the sensor anomaly judgment interval as [c, d];

[0055] S3: Obtain the duration of the current task execution anomaly. Compare the duration with the traction anomaly judgment interval and the sensor anomaly judgment interval. Based on the comparison results, selectively remove image frames from the monitored video: When the rescue vehicle is actually performing the task, collect and analyze the weight data sensed by the weight sensor. If the weight data changes from decreasing to increasing, the current task execution is judged to be abnormal. The time point when the weight starts to decrease is V1, and the time point when the weight starts to increase after decreasing is V2. The duration of the current task execution anomaly is V2-V1. If V2-V1 is within the interval [a,b], the cause of the current task execution anomaly is judged to be the backward movement of the faulty vehicle due to traction anomaly during loading. Image frames from the monitored video during the time period from V1 to V2 are removed. If V2-V1 is within the interval [c,d], the cause of the current task execution anomaly is judged to be the sensor failure of the rescue vehicle. No image frame removal is performed. If V2-V1 is not within either of these two intervals, further investigation is conducted to determine the cause of the rescue vehicle's task execution anomaly.

[0056] S4: After the elimination process is completed, the empty / heavy load status identification and operation event identification of the rescue vehicle are performed: After the elimination process is completed, the monitored video is transmitted to the vehicle-mounted edge device. The vehicle-mounted edge device preprocesses each frame of the video, calibrates the flat platform area of ​​the rescue vehicle, and uses the improved YOLOv5 algorithm to identify whether there are faulty vehicles in the calibrated area. A detection time is set, and the number of image frames containing faulty vehicles and the number of image frames without faulty vehicles identified within the detection time are counted. A frame count threshold is set. If the number of image frames containing faulty vehicles exceeds the frame count threshold, the rescue vehicle is judged to be in a heavy load state; if the number of image frames without faulty vehicles exceeds the frame count threshold, the rescue vehicle is judged to be in an empty load state. If the rescue vehicle is identified as empty in the previous detection time and as heavy load in the next detection time, the rescue vehicle is judged to be loading cargo; if the rescue vehicle is identified as heavy load in the previous detection time and as empty in the next detection time, the rescue vehicle is judged to be unloading cargo.

[0057] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A rescue vehicle empty / loaded status recognition system based on multi-source data, characterized in that: The system includes: a sensor information acquisition module, a sensor data mapping and analysis module, an image frame elimination and selection module, and a state recognition module; The sensor information acquisition module collects weight sensing data and monitoring video data from the rescue vehicle's historical missions. The sensor data mapping and analysis module filters out historical videos of the rescue vehicle loading, confirms the historical loading time interval, arranges and analyzes the historical weight data sensed during the loading process, generates a line graph of historical sensor data, filters out abnormal line graphs, and sets traction abnormal judgment intervals and sensor abnormal judgment intervals. The image frame elimination selection module obtains the duration of the current task execution anomaly, compares the duration of the current task execution anomaly with the traction anomaly judgment interval and the sensor anomaly judgment interval respectively, and performs selective elimination processing on the image frames of the monitored video based on the comparison results. The status recognition module identifies the empty / loaded status of the rescue vehicle and the operation event after the elimination process is completed. The image frame elimination selection module includes: a current data analysis unit and an image frame elimination selection unit; The current data analysis unit is used to collect and analyze the weight data sensed by the weight sensor when the rescue vehicle is actually performing a task. If the weight data changes from decreasing to increasing, it is determined that the current task is abnormal. The unit also counts the time when the weight data starts to decrease and the time when it starts to increase after decreasing, and obtains the duration of the current task execution abnormality. The image frame rejection selection unit is used to compare the duration of the current task execution anomaly with the traction anomaly judgment interval and the sensor anomaly judgment interval, respectively. If the duration of the current task execution anomaly is within the traction anomaly judgment interval, the cause of the current task execution anomaly is determined to be the backward movement of the malfunctioning vehicle due to traction anomaly during loading. If the duration of the current task execution anomaly is within the sensor anomaly judgment interval, the cause of the current task execution anomaly is determined to be the sensor malfunction of the rescue vehicle. If the cause of the current task execution anomaly is the backward movement of the malfunctioning vehicle due to traction anomaly during loading, the image frames of the video monitored within the current task execution anomaly time interval will be rejected. Otherwise, no image frame rejection processing will be performed.

2. The rescue vehicle empty / heavy load status identification system based on multi-source data according to claim 1, characterized in that: The sensing information acquisition module includes: a weight sensing data acquisition unit and a historical monitoring data acquisition unit; The weight sensing data acquisition unit is used to collect historical weight data sensed by the rescue vehicle during previous missions. The historical monitoring data acquisition unit is used to collect historical video data captured by the cameras installed on the rescue vehicle.

3. The rescue vehicle empty / heavy load status identification system based on multi-source data according to claim 2, characterized in that: The sensing data mapping and analysis module includes: a sensing data arrangement unit, a line chart generation and analysis unit, and an anomaly judgment interval setting unit; The sensing data arrangement unit is used to filter out video data of the rescue vehicle loading from historical video data, confirm the start time and end time of the filtered video, filter out the historical weight data sensed from the start time to the end time, and combine the filtered historical weight data and sensing time data into data points and arrange them in a rectangular coordinate system. The horizontal axis of the rectangular coordinate system represents the weight data sensing time, the vertical axis represents the historical weight data, and the weight data sensing time is distributed at equal intervals. The line chart generation and analysis unit is used to connect adjacent data points and generate several historical sensing data line charts. Each line chart is analyzed: it is determined whether the slope between adjacent data points is negative. If it is, the change feature of the adjacent data points is recorded as feature value 0; otherwise, it is recorded as feature value 1. After the analysis of a line chart is completed, a feature column matrix is ​​generated based on the statistical feature values. A benchmark judgment column matrix is ​​set. Based on the feature column matrix and the benchmark judgment column matrix, it is determined whether the corresponding line chart is an abnormal line chart, and all abnormal line charts are filtered out. The anomaly judgment interval setting unit is used to obtain the abnormal time interval in each abnormal line graph, retrieve the video monitored within the abnormal time interval, and determine whether the faulty vehicle being towed by the rescue vehicle during loading in the video exhibits a backward movement. Based on the judgment result, the abnormal line graph is divided into a first abnormal line graph and a second abnormal line graph. The first abnormal line graph is a traction abnormal data line graph, and the second abnormal line graph is a sensor abnormal data line graph. Anomalies in the first abnormal line graph are obtained, and the difference in horizontal coordinates between the anomaly with the smallest and largest horizontal coordinates is confirmed. This difference in horizontal coordinates is used as the traction abnormality duration, resulting in several traction abnormality durations. Similarly, anomalies in the second abnormal line graph are obtained, and the difference in horizontal coordinates between the anomaly with the smallest and largest horizontal coordinates is confirmed. This difference in horizontal coordinates is used as the sensor abnormality duration, resulting in several sensor abnormality durations. The traction abnormality judgment interval and the sensor abnormality judgment interval are set based on these traction abnormality durations and sensor abnormality durations.

4. The rescue vehicle empty / loaded status identification system based on multi-source data according to claim 1, characterized in that: The status recognition module includes: a video stream transmission unit, an idle / overload status recognition unit, and a job event analysis unit; The video stream transmission unit is used to transmit the monitored video to the vehicle edge device after the elimination process is completed, and the vehicle edge device is used to preprocess each frame of the video. The empty / heavy load state identification unit is used to calibrate the flatbed platform area of ​​the rescue vehicle. It uses an improved YOLOv5 algorithm to identify whether there are faulty vehicles in the calibrated area. It sets a detection time, counts the number of image frames containing faulty vehicles and the number of image frames without faulty vehicles identified within the detection time, and sets a frame count threshold. If the number of image frames containing faulty vehicles exceeds the frame count threshold, it is determined that the rescue vehicle is in a heavy load state; if the number of image frames without faulty vehicles exceeds the frame count threshold, it is determined that the rescue vehicle is in an empty load state. The operation event analysis unit is used to identify rescue vehicle operation events: if the rescue vehicle is identified as empty in the previous detection period and as heavily loaded in the next detection period, it is determined that the rescue vehicle is loading; if the rescue vehicle is identified as heavily loaded in the previous detection period and as empty in the next detection period, it is determined that the rescue vehicle is unloading.

5. A method for identifying the empty / loaded status of a rescue vehicle based on multi-source data, characterized in that: Includes the following steps: S1: Collect weight sensor data and monitoring video data from the rescue vehicle's historical missions; S2: Filter out historical videos of the rescue vehicle loading, confirm the historical loading time range, arrange and analyze the historical weight data sensed during the loading process, generate a line chart of historical sensing data, filter out abnormal line charts, and set the traction abnormal judgment interval and sensor abnormal judgment interval. S3: Obtain the duration of the current task execution anomaly, compare the duration of the current task execution anomaly with the traction anomaly judgment interval and the sensor anomaly judgment interval respectively, and selectively remove the image frames of the monitored video based on the comparison results. S4: After the elimination process is completed, identify the empty and heavy load status of the rescue vehicle and identify the operation event; In step S3: When the rescue vehicle is actually performing the task, the weight data sensed by the weight sensor is collected and analyzed. If the weight data changes from decreasing to increasing, it is determined that the current task execution is abnormal. The time point when the weight starts to decrease is V1 and the time point when the weight starts to increase after decreasing is V2. The duration of the current task execution abnormality is V2-V1. If V2-V1 is within the traction abnormality judgment interval, it is determined that the reason for the current task execution abnormality is that the faulty vehicle reversed due to traction abnormality during loading. The image frames of the video monitored during the time period from V1 to V2 are removed. If V2-V1 are within the sensor anomaly detection range, the cause of the current task execution anomaly is determined to be a sensor malfunction of the rescue vehicle, and no image frame removal processing is performed.

6. The method for identifying the empty / heavy load status of a rescue vehicle based on multi-source data according to claim 5, characterized in that: In step S2: Historical loading videos of the rescue vehicle are selected from the monitoring video data during a random mission. The start time of the selected videos is T1, and the end time is T2. The time interval of a random historical loading is confirmed as [T1, T2]. Historical weight data sensed within the time interval [T1, T2] are selected to obtain the historical weight set M = {M1, M2, ... M}. k The time it takes for the weight sensor to detect the corresponding weight is t = {t1, t2, ..., t}. k Within the time interval [T1, T2], a total of k data points were sensed, which were combined into data points {(t1, M1), (t2, M2), ..., (t...}. k M k Arrange the data points in a rectangular coordinate system, connect adjacent data points to generate a line graph of historical sensing data, and calculate the slope between the first group of adjacent data points as F, F=(M2-M1) / (t2-t1). If F is negative, record the change characteristic of the first group of adjacent data points as feature value 0; if F is not negative, record it as feature value 1. Determine whether the slope between all adjacent data points is negative, and generate a feature value set {A1,A2,...,A1}. k-1 The generated feature column matrix is: Set a benchmark column matrix Given that all elements in the baseline judgment column matrix are 1, and there are a total of k-1 elements, determine whether the corresponding line chart is an abnormal line chart according to the following method: ； in, This is the anomaly detection matrix for the line chart. If the matrix is ​​all zeros, then the corresponding line chart is determined not to be an abnormal line chart; otherwise, the corresponding line chart is determined to be an abnormal line chart. From the monitoring video data during the n missions of the rescue vehicle, n historical videos of the rescue vehicle loading are selected, n historical sensing data line charts are generated, and all abnormal line charts are selected.

7. The method for identifying the empty / heavy load status of a rescue vehicle based on multi-source data according to claim 6, characterized in that: Obtain the abnormal time interval in each abnormal line graph, retrieve the video monitored in the abnormal time interval, and determine whether the faulty vehicle being towed by the rescue vehicle during loading in the video has a backward movement. If a backward movement is observed, classify the abnormal line graph as the first abnormal line graph. Otherwise, the abnormal line chart is divided into a second abnormal line chart. The first abnormal line chart is the traction abnormal data line chart, and the second abnormal line chart is the sensor abnormal data line chart. A total of m first abnormal line charts are counted. Abnormal points in the first abnormal line charts are obtained. If the slope between adjacent data points is negative, then the corresponding two adjacent data points are abnormal points. The difference in the horizontal coordinate between the abnormal point with the smallest horizontal coordinate and the abnormal point with the largest horizontal coordinate is identified. The difference in the horizontal coordinate is used as the duration of traction abnormality. The durations of m traction abnormalities are obtained as W={W1,W2,...,W... m }, a total of r second-anomaly line graphs were counted, and the duration of the r sensor anomalies was obtained as w={w1,w2,...,w r }, we obtain the value that appears most frequently and is the smallest in set W as a, and the value that appears most frequently and is the largest as b, and set the traction anomaly judgment interval as [a, b]; we obtain the value that appears most frequently and is the smallest in set w as c, and the value that appears most frequently and is the largest as d, and set the sensor anomaly judgment interval as [c, d].

8. The method for identifying the empty / heavy load status of a rescue vehicle based on multi-source data according to claim 7, characterized in that: In step S4: After the elimination process is completed, the monitored video is transmitted to the vehicle-mounted edge device. The vehicle-mounted edge device preprocesses each frame of the video, calibrates the flat platform area of ​​the rescue vehicle, and uses the improved YOLOv5 algorithm to identify whether there are faulty vehicles in the calibrated area. A detection duration is set, and the number of image frames containing faulty vehicles and the number of image frames without faulty vehicles identified within the detection duration are counted. A frame count threshold is set. If the number of image frames containing faulty vehicles exceeds the frame count threshold, the rescue vehicle is determined to be in a heavily loaded state; if the number of image frames without faulty vehicles exceeds the frame count threshold, the rescue vehicle is determined to be in an unloaded state. If the rescue vehicle is identified as unloaded in the previous detection duration and as heavily loaded in the next detection duration, the rescue vehicle is determined to be loading cargo; if the rescue vehicle is identified as heavily loaded in the previous detection duration and as unloaded in the next detection duration, the rescue vehicle is determined to be unloading cargo.

Images