Vehicle operating status monitoring method, system, device and storage medium

Abstract

The present invention discloses a vehicle operation status monitoring method, system, device and storage medium. The method includes: obtaining panoramic image information from the interior of a target vehicle, and obtaining image information of the driver's foot movements; obtaining the driver's driving operation data based on the panoramic image information and the foot movement image information; obtaining the target vehicle's vehicle monitoring data, and judging whether the target vehicle has an abnormality based on the vehicle monitoring data and the driving operation data; when it is determined that the target vehicle has an abnormality, terminating the target vehicle's driving signal and uploading insurance information. The present invention identifies and integrates the panoramic image information and the foot movement image information to obtain the driver's driving operation data, and judges whether the vehicle has an abnormality in real time based on the driving operation data and the vehicle monitoring data. When the vehicle has an abnormality, the vehicle can be braked and insurance information can be automatically uploaded, thereby ensuring the user's driving safety and improving the user's driving experience. The invention can be widely used in the field of vehicle monitoring technology.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle monitoring, and in particular to a vehicle operating status monitoring method, system, device and storage medium. Background Art

[0002] With the rapid development of automobile electrification, brake failure and drive motor stall incidents occur frequently, making post-accident investigations and accountability difficult. Currently, various monitoring devices in the vehicle are independent of the vehicle computer system, and only the monitoring data is retained without integrating resources for real-time vehicle anomaly detection, affecting users' driving safety and experience. Summary of the Invention

[0003] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0004] To this end, an object of an embodiment of the present invention is to provide a vehicle operating status monitoring method, which can determine in real time whether an abnormality occurs in the vehicle, thereby ensuring the user's driving safety and improving the user's driving experience.

[0005] Another object of an embodiment of the present invention is to provide a vehicle operating status monitoring system.

[0006] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:

[0007] In a first aspect, an embodiment of the present invention provides a method for monitoring a vehicle operating state, comprising the following steps:

[0008] Obtain panoramic image information of the target vehicle interior and obtain image information of the driver's foot movements;

[0009] obtaining the driver's driving operation data based on the panoramic image information and the foot movement image information;

[0010] Acquiring vehicle-mounted monitoring data of the target vehicle, and determining whether an abnormality occurs with the target vehicle based on the vehicle-mounted monitoring data and the driving operation data;

[0011] When it is determined that an abnormality occurs in the target vehicle, the driving signal of the target vehicle is terminated and the insurance report information is uploaded.

[0012] Furthermore, in one embodiment of the present invention, the step of obtaining panoramic image information of the interior of the target vehicle and obtaining image information of the driver's foot movements is specifically as follows:

[0013] Acquire the panoramic image information by shooting in real time with a panoramic camera device, and acquire the foot movement image information by shooting in real time with a foot camera device;

[0014] Wherein, the panoramic camera device is installed above the driver's seat of the target vehicle, and the foot camera device is installed at the bottom of the driver's seat of the target vehicle.

[0015] Furthermore, in one embodiment of the present invention, the step of identifying and obtaining the driver's driving operation data based on the panoramic image information and the foot movement image information specifically includes:

[0016] determining hand motion image information of the driver based on the panoramic image information, determining a plurality of continuous hand motion image frames based on the hand motion image information, and further determining hand motion time series data based on the hand motion image frames;

[0017] determining a plurality of continuous foot motion image frames according to the foot motion image information, and determining foot motion time series data according to the foot motion image frames;

[0018] Inputting the hand motion time series data into a pre-trained hand driving operation recognition model to obtain the driver's hand driving operation data, and inputting the foot motion time series data into a pre-trained foot driving operation recognition model to obtain the driver's foot driving operation data;

[0019] The driving operation data is determined based on the hand driving operation data and the foot driving operation data.

[0020] Furthermore, in one embodiment of the present invention, the vehicle operation status monitoring method further includes the steps of pre-training the hand driving operation recognition model and the foot teaching operation recognition model, which specifically include:

[0021] Obtaining a first training sample and a second training sample, wherein the first training sample is time-series data of hand movements of a test person during a test drive, and the second training sample is time-series data of foot movements of the test person during a test drive;

[0022] Manually labeling the first training samples to obtain hand driving operation labels, and constructing a first training dataset based on the first training samples and the corresponding hand driving operation labels;

[0023] Manually labeling the second training samples to obtain foot driving operation labels, and constructing a second training data set based on the second training samples and the corresponding foot driving operation labels;

[0024] Inputting the first training data set into a pre-built first long short-term memory neural network for training, and optimizing the model parameters of the first long short-term memory neural network using a back propagation algorithm to obtain a trained hand driving operation recognition model;

[0025] The second training data set is input into a pre-built second long short-term memory neural network for training, and the model parameters of the second long short-term memory neural network are optimized using a back propagation algorithm to obtain a trained foot driving operation recognition model.

[0026] Furthermore, in one embodiment of the present invention, the step of obtaining the vehicle-mounted monitoring data of the target vehicle and determining whether an abnormality occurs in the target vehicle based on the vehicle-mounted monitoring data and the driving operation data specifically includes:

[0027] Acquiring the vehicle monitoring data through a vehicle monitoring system, wherein the vehicle monitoring data includes a vehicle driving / braking state, a vehicle steering state, and a vehicle gear state;

[0028] Comparing the vehicle-computer monitoring data with the driving operation data to determine whether the vehicle-computer monitoring data corresponds to the driving operation data;

[0029] When the vehicle-computer monitoring data does not correspond to the driving operation data, it is determined that an abnormality occurs in the target vehicle; when the vehicle-computer monitoring data corresponds to the driving operation data, it is determined that no abnormality occurs in the target vehicle.

[0030] Furthermore, in one embodiment of the present invention, when it is determined that the target vehicle has an abnormality, the step of terminating the driving signal of the target vehicle and uploading the insurance report information is specifically as follows:

[0031] When it is determined that an abnormality occurs in the target vehicle, the motor power supply of the target vehicle is cut off, and an insurance report is generated based on the vehicle monitoring data and the driving operation data, and then the insurance report is uploaded through the evidence device of the target vehicle and the user terminal of the driver.

[0032] Furthermore, in one embodiment of the present invention, the vehicle operating status monitoring method further includes the following steps:

[0033] The data exchange instantaneous flow rate and the data exchange IP address are determined by the vehicle computer monitoring data, and whether there is a hacker intrusion is determined based on the data exchange instantaneous flow rate and the data exchange IP address.

[0034] In a second aspect, an embodiment of the present invention provides a vehicle operation status monitoring system, comprising:

[0035] An image information acquisition module is used to acquire panoramic image information of the target vehicle and the driver's foot movement image information;

[0036] a driving operation recognition module, configured to obtain the driver's driving operation data based on the panoramic image information and the foot movement image information;

[0037] A vehicle status judgment module is used to obtain the vehicle monitoring data of the target vehicle and judge whether the target vehicle has any abnormality based on the vehicle monitoring data and the driving operation data;

[0038] The insurance reporting module is used to terminate the driving signal of the target vehicle and upload the insurance reporting information when it is determined that the target vehicle has an abnormality.

[0039] In a third aspect, an embodiment of the present invention provides a vehicle operating status monitoring device, comprising:

[0040] at least one processor;

[0041] at least one memory for storing at least one program;

[0042] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned vehicle operating status monitoring method.

[0043] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing a program executable by a processor, wherein the program executable by the processor is used to execute the above-mentioned vehicle operation status monitoring method when executed by the processor.

[0044] The advantages and benefits of the present invention will be described in part in the following description and will become apparent from the following description or learned through practice of the present invention:

[0045] The embodiment of the present invention obtains the panoramic image information inside the target vehicle and the image information of the driver's foot movements, identifies the driver's driving operation data based on the panoramic image information and the foot movement image information, then obtains the vehicle monitoring data of the target vehicle, determines whether the target vehicle has an abnormality based on the vehicle monitoring data and the driving operation data, and when it is determined that the target vehicle has an abnormality, terminates the driving signal of the target vehicle and uploads the insurance report information. The embodiment of the present invention can obtain the driver's driving operation data by identifying and integrating the panoramic image information inside the vehicle and the driver's foot movement image information, and determines whether the vehicle has an abnormality in real time based on the driving operation data and the vehicle monitoring data. When the vehicle has an abnormality, the vehicle can be braked and the insurance report information can be automatically uploaded, thereby ensuring the user's driving safety and improving the user's driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 A flowchart of a method for monitoring vehicle operating status provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of the sequential structure of a long short-term memory neural network repeating unit provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of gate control of a long short-term memory neural network provided by an embodiment of the present invention;

[0050] Figure 4 A structural block diagram of a vehicle operating status monitoring system provided by an embodiment of the present invention;

[0051] Figure 5 This is a structural block diagram of a vehicle operating status monitoring device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] In the description of the present invention, "a plurality" means two or more. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly indicating the number of the indicated technical features, or as implicitly indicating the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art.

[0054] Reference Figure 1 The embodiment of the present invention provides a vehicle operation status monitoring method, which specifically includes the following steps:

[0055] S101: Obtain panoramic image information of the interior of a target vehicle and obtain image information of the driver's foot movements.

[0056] Specifically, an embodiment of the present invention obtains panoramic image information inside the vehicle and image information of the driver's foot movements to facilitate subsequent identification of driving operations. It can be understood that the panoramic image information records the driver's hand operations, such as turning the steering wheel, shifting the gear lever, etc., and the foot movement image information records the driver's foot operations, such as stepping on the brakes, stepping on the accelerator, etc.

[0057] As an optional embodiment, step S101 of obtaining panoramic image information of the interior of the target vehicle and obtaining image information of the driver's foot movements is specifically as follows:

[0058] Acquire panoramic image information by real-time shooting with a panoramic camera device, and acquire foot movement image information by real-time shooting with a foot camera device;

[0059] The panoramic camera device is installed above the driver's seat of the target vehicle, and the foot camera device is installed at the bottom of the driver's seat of the target vehicle.

[0060] S102 : Obtaining the driver's driving operation data based on the panoramic image information and the foot movement image information.

[0061] Specifically, the panoramic image information includes continuous image frames of the driver's hand operations, and the foot movement image information includes continuous image frames of the driver's foot movements. By identifying these continuous image frames, the driver's hand and foot driving operations can be obtained, and thus the complete driving operation data can be integrated. Step S102 specifically includes the following steps:

[0062] S1021: Determine hand motion image information of the driver based on the panoramic image information, determine a plurality of continuous hand motion image frames based on the hand motion image information, and further determine hand motion time series data based on the hand motion image frames;

[0063] S1022, determining a plurality of continuous foot motion image frames according to the foot motion image information, and determining foot motion time series data according to the foot motion image frames;

[0064] S1023. Input the hand motion time series data into a pre-trained hand driving operation recognition model to obtain the driver's hand driving operation data. Input the foot motion time series data into a pre-trained foot driving operation recognition model to obtain the driver's foot driving operation data.

[0065] S1024. Determine driving operation data based on the hand driving operation data and the foot driving operation data.

[0066] Specifically, an embodiment of the present invention uses a long short-term memory neural network to pre-train to obtain a hand driving operation recognition model and a foot driving operation recognition model. The hand operation recognition model can recognize the hand driving operation based on the temporal changes of the image features of each image frame in the hand movement timing data, and the foot operation recognition model can recognize the foot driving operation based on the temporal changes of the image features of each image frame in the foot movement timing data.

[0067] As an optional embodiment, the vehicle operation status monitoring method further includes the steps of pre-training a hand driving operation recognition model and a foot teaching operation recognition model, which specifically include:

[0068] A1. Obtain a first training sample and a second training sample. The first training sample is the time series data of the test person's hand movements during the test drive, and the second training sample is the time series data of the test person's foot movements during the test drive.

[0069] A2. Manually label the first training sample to obtain hand driving operation labels, and construct a first training dataset based on the first training sample and the corresponding hand driving operation labels;

[0070] A3. Manually label the second training samples to obtain foot driving operation labels, and construct a second training dataset based on the second training samples and the corresponding foot driving operation labels;

[0071] A4. Inputting the first training data set into a pre-built first long short-term memory neural network for training, optimizing the model parameters of the first long short-term memory neural network using a back propagation algorithm to obtain a trained hand driving operation recognition model;

[0072] A5. Input the second training data set into the pre-built second long short-term memory neural network for training, and optimize the model parameters of the second long short-term memory neural network using a back propagation algorithm to obtain a trained foot driving operation recognition model.

[0073] Specifically, when constructing the training data set, the driving operation label corresponding to each training sample is determined based on the actual driving operation of the test person. For example, if the test person turns the steering wheel to the left during the test drive, the corresponding hand driving operation label of the hand action time series data can be "turn the steering wheel left". For another example, if the test person steps on the accelerator during the test drive, the corresponding foot driving operation label of the foot action time series data can be "step on the accelerator", and so on.

[0074] In some optional embodiments, the label may be further refined according to the magnitude of the action, such as lightly pressing the accelerator, heavily pressing the accelerator, etc., which will not be elaborated in detail in the embodiments of the present invention.

[0075] A sufficient amount of data from different testers and different driving operations is collected as the first training sample and the second training sample to form the first training sample set and the second training sample set.

[0076] The following describes the long short-term memory neural network and the model training process according to an embodiment of the present invention.

[0077] Long Short Term Memory Network (LSTM) is an improved recurrent neural network. In the LSTM network, conventional neurons are replaced by storage units, and each storage unit consists of an input gate, an output gate, a unit state, etc.

[0078] The sequential structure of the LSTM network repetitive units is as follows Figure 2 As shown, at time t, the LSTM has three inputs: the input value X of the network at the current moment t , the output value X of LSTM at the previous moment t-1 , and the unit state C at the previous moment t-1 LSTM has two outputs: the current LSTM output value h t , and the current cell state C t .

[0079] For the control of the long-term state c, three gate control switches are used for measurement. It is necessary to consider the influence of the output of the previous storage unit, the influence of the current input value, and the influence of the current storage unit on subsequent transmission. Therefore, the rational use of the three gate control switches becomes an important link in signal transmission and parameter update.

[0080] In the process of transferring information over time, three control switches are used to measure the state unit c. The specific allocation is as follows: Figure 3 As shown, the first switch (left) controls the continued storage of the long-term state c, measuring the influence of the previous storage unit on the current storage unit and determining the impact of memory experience on the current time point. The second switch (bottom) controls the input of the immediate state into the long-term state c. The input signal does not need to be passed to the model in full; only the measurement values ​​that need to be considered are passed. The third switch (right) controls whether the long-term state c is used as the output of the current LSTM. It is worth noting that the gate switch used here is only a gate definition for its function. Its value is a vector between 0 and 1, and it is calculated using a weighted metric, rather than a gate switch that is simply considered to be 0 or 1.

[0081] A gate switch is essentially a fully connected layer. Its input is a vector, and its output is a real number vector between 0 and 1. To use it, the gate switch output vector is element-wise multiplied by the vector to be controlled. The gate output is a real number vector between 0 and 1. When the gate output is 0, multiplying any vector with it yields a 0 vector, meaning nothing passes through. When the gate output is 1, multiplying any vector with it yields no change, meaning everything passes through. The sigmoid function normalizes the output, giving the output a probability value between 0 and 1.

[0082] The embodiment of the present invention uses the test person's hand movement timing data / foot movement timing data as the input data of the LSTM model to train the network. This process can perform layer-by-layer feature learning and mapping on the data, and then send the deep time-frequency feature signal features into the SoftMax classifier for training. Then, the LSTM is initialized using the trained weights, and the constructed LSTM network model is optimized using the back propagation algorithm, so that the LSTM network model converges to reach the global optimum, and finally achieves the purpose of face turning recognition.

[0083] As an optional implementation, the model parameters of the long short-term memory neural network include a forget gate weight matrix, an input gate weight matrix, an output gate weight matrix, a unit state weight matrix, a forget gate bias term, an input gate bias term, an output gate bias term, and a unit state bias term. The step of optimizing the model parameters of the long short-term memory neural network using a backpropagation algorithm specifically includes:

[0084] B1. Forward calculation of the forget gate, input gate, output gate and unit state of each storage unit;

[0085] B2. Reversely calculate the error term value of each storage unit based on the forget gate, input gate, output gate and unit state;

[0086] B3. Determine the first gradient of the forget gate weight matrix, the second gradient of the input gate weight matrix, the third gradient of the output gate weight matrix, and the fourth gradient of the unit state weight matrix according to the error term value;

[0087] B4. Perform gradient updates on the first gradient, the second gradient, the third gradient, and the fourth gradient, thereby optimizing the forget gate weight matrix, the input gate weight matrix, the output gate weight matrix, the unit state weight matrix, the forget gate bias term, the input gate bias term, the output gate bias term, and the unit state bias term.

[0088] Specifically, when constructing a long short-term memory neural network, in addition to determining the initial model parameters, it is also necessary to determine the forget gate function, input gate function, output gate function, and unit state function.

[0089] The forget gate function is:

[0090] f t =σ(W f [h t-1 ,x t ]+b f )

[0091] Among them, f t represents the forget gate, W f represents the forget gate weight matrix, [h t-1 ,x t ] represents the input signal of the previous layer output and the current layer input as a whole, b f represents the forget gate bias term, σ represents the sigmoid function;

[0092] The input gate function is:

[0093] i t =σ(W i [h t-1 ,x t ]+b i )

[0094] Among them, i t represents the input gate, W i represents the input gate weight matrix, [h t-1 , x t ] represents the input signal of the previous layer output and the current layer input as a whole, b i Represents the input gate bias term, σ represents the sigmoid function;

[0095] The output gate function is:

[0096] o t =σ(W o .[h t-1 , x t ]+b o )

[0097] Among them, t represents the output gate, W o represents the output gate weight matrix, [h t-1 , x t ] represents the input signal of the previous layer output and the current layer input as a whole, b o Represents the output gate bias term, σ represents the sigmoid function;

[0098] The cell state function is:

[0099] C t =f t *C t-1 +it *C′ t

[0100] C′ t =tanh(W c .[h t-1 , x t ]+b c )

[0101] Among them, C t Indicates the current unit state, f t represents the forget gate, C t-1 Indicates the unit state at the previous moment, i t represents the input gate, C′ t Represents the current input unit state, tanh represents the tanh function, W c represents the unit state weight matrix, [h t-1 , x t ] represents the input signal of the previous layer output and the current layer input as a whole, b c Represents the unit state bias term.

[0102] During the forward transmission process, the signal transmission process is controlled by controlling the gate control switches between adjacent storage units. It can be mainly divided into three categories, as follows:

[0103] 1) Forget gate: determines the cell state C at the previous moment t-1 How much information is still retained at the current moment C t In. Implement memory update of data.

[0104] 2) Input gate: determines the input X of the network at the current moment t How much is saved to the cell state C t , to achieve the screening of input data and reduce the purpose of data input.

[0105] 3) Output gate: controls the cell state C t How many outputs are given to the LSTM's current output value h? t , reducing data output.

[0106] The forget gate can save information from a long time ago, normalize the output value through the σ function, and use the influence of the original information as an important reference to determine the propagation process of the current value, thereby achieving the purpose of memory updating.

[0107] The input gate can filter the current input data to prevent currently irrelevant content from entering the storage unit.

[0108] The unit state of the current input is calculated based on the previous output and this input; the unit state C at the current moment t Calculation: Based on the last cell state C t-1 Multiply the forget gate f element-wise t , and then use the current input unit state C t Element-wise multiplication of the input gate i t , then add the two products: In this way, the current memory C t and long-term memory C t-1 Combined together, a new unit state C is formed t Due to the control of the forget gate, it can save information from a long time ago, and due to the control of the input gate, it can prevent currently irrelevant content from entering the storage unit.

[0109] The main function of the output gate is to control the influence of long-term memory on the current output. The output value h of the output gate t as follows:

[0110] h t =o t *tanh(C t )

[0111] Similar to the BP neural network, the reverse update of the LSTM network can be divided into three steps: determining parameters, calculating errors, and updating gradients.

[0112] 1) Determine the parameters: forward calculate the output value of each storage unit, a total of 4 variables (f t 、i t , c′ t 、o t ), the calculation method is shown in the above specific formula.

[0113] 2) Error Calculation: Reversely calculate the error term value of each storage unit. Like RNN, the backpropagation of LSTM error terms also includes two directions: one is backpropagation along time, that is, starting from the current time t, calculating the error term at each moment; the other is propagating the error term to the previous layer.

[0114] 3) Gradient update: Calculate the gradient of each weight based on the corresponding error term.

[0115]

[0116] σ′(z)=y(1-y)

[0117] The activation function of the door switch is defined as the sigmoid function, the output activation function is the tanh function, and the derivatives are:

[0118]

[0119] tanh′(z)=1-y 2

[0120] The goal of network optimization is to learn 8 sets of parameters, including the weight matrix and bias term of the forget gate, the weight matrix and bias term of the input gate, the weight matrix and bias term of the output gate, and the weight matrix and bias term of the unit state.

[0121] The weight matrix W is composed of two matrices. The two parts use different formulas in the back propagation. In the subsequent derivation, the weight matrix should also be written as two separate matrices [w fh , w fx ].

[0122] In the reverse propagation of the error term along time, the error term at time t-1 is calculated as follows:

[0123]

[0124] Using h t and C t The definition of and the total derivative formula can be used to obtain the formula for passing the error term forward to any k moment:

[0125]

[0126] The gradient update is shown in Table 1 below.

[0127]

[0128] Table 1

[0129] This embodiment of the present invention uses a backpropagation algorithm to optimize and update the model parameters of the long short-term memory neural network. After several iterations, a trained hand / foot driving operation recognition model is obtained. The specific number of iterations can be pre-set, or training is considered complete when the test set meets the required accuracy.

[0130] For the hand operation timing data and foot operation timing data to be identified, they are respectively input into the trained hand driving operation recognition model and foot driving operation recognition model to obtain the driver's hand driving operation data and foot driving operation data at each moment. By integrating them, the driver's complete driving operation data can be obtained.

[0131] S103: Obtain vehicle-mounted computer monitoring data of the target vehicle, and determine whether the target vehicle has any abnormality based on the vehicle-mounted computer monitoring data and the driving operation data.

[0132] Specifically, the vehicle monitoring data can be obtained through the vehicle monitoring system, mainly including the vehicle driving / braking status, vehicle steering status, vehicle gear status, etc., while the driving operation data includes the driver's operation data on the brake, accelerator, steering wheel and gear lever. By comparing the driving operation data with the various status parameters of the vehicle, it can be determined whether the target vehicle has any abnormality. Step S103 specifically includes the following steps:

[0133] S1031. Acquire vehicle monitoring data through the vehicle monitoring system, where the vehicle monitoring data includes vehicle driving / braking status, vehicle steering status, and vehicle gear status;

[0134] S1032, comparing the vehicle computer monitoring data with the driving operation data to determine whether the vehicle computer monitoring data corresponds to the driving operation data;

[0135] S1033: When the vehicle-computer monitoring data does not correspond to the driving operation data, it is determined that an abnormality has occurred in the target vehicle; when the vehicle-computer monitoring data corresponds to the driving operation data, it is determined that no abnormality has occurred in the target vehicle.

[0136] Specifically, when the vehicle driving / braking state is the driving state and the driving operation data at the corresponding moment is stepping on the accelerator, it means that the two correspond. Similarly, when the vehicle steering state is left turn and the driving operation data at the corresponding moment is turning the steering wheel left, it means that the two correspond. When the vehicle gear state is P gear and the driving operation data at the corresponding moment is moving the gear lever to P gear, it means that the two correspond.

[0137] When the vehicle monitoring data does not correspond to the driving operation data, it indicates that there is an abnormality in the target vehicle, such as stalling, abnormal braking, etc.

[0138] S104: When it is determined that an abnormality occurs in the target vehicle, the driving signal of the target vehicle is terminated and the insurance report information is uploaded.

[0139] As an optional embodiment, when it is determined that the target vehicle has an abnormality, the driving signal of the target vehicle is terminated and the insurance report information is uploaded in step S104, which is specifically as follows:

[0140] When it is determined that the target vehicle has an abnormality, the motor power of the target vehicle is cut off, and an insurance report is generated based on the vehicle monitoring data and driving operation data, and then the insurance report is uploaded through the evidence device of the target vehicle and the driver's user terminal.

[0141] Specifically, when it is determined that an abnormality has occurred in the target vehicle, the motor power of the target vehicle can be automatically cut off, or an alarm can be issued for the driver to manually cut off the motor power of the target vehicle; corresponding insurance report information can be generated based on the comparison results of the vehicle monitoring data and the driving operation data, and at the same time, evidence information can be obtained through the target vehicle's radar detection device and drone camera device, and then uploaded to the insurance company and transportation department through the driver's user terminal.

[0142] Furthermore, the acquired panoramic image information, foot movement image information, and vehicle monitoring data must all be uploaded to the cloud for subsequent retrospective query. Furthermore, the identification of hand and foot driving operations and the comparison of driving operation data with vehicle monitoring data can be performed in the cloud, reducing the computing load on the vehicle. The vehicle system uploads the panoramic image information, foot movement image information, and vehicle monitoring data to the cloud device through data exchange.

[0143] As an optional embodiment, the vehicle operation status monitoring method further includes the following steps:

[0144] The vehicle computer monitoring data is used to determine the instantaneous data exchange traffic and the data exchange IP address, and based on the instantaneous data exchange traffic and the data exchange IP address, it is determined whether there is a hacker intrusion.

[0145] Specifically, the instantaneous data exchange traffic and the data exchange IP address can be used to determine whether the vehicle system has been invaded. If the instantaneous data exchange traffic suddenly increases when the car owner is not actively downloading, or the data exchange IP address is not the network address specified by the car owner, it means that the vehicle system may have been invaded, causing vehicle abnormalities. At this time, the instantaneous data exchange traffic and the data exchange IP address can be used as evidence information for reporting insurance.

[0146] The above describes the method steps of an embodiment of the present invention. It is understood that the embodiment of the present invention can obtain the driver's driving operation data by identifying and integrating the panoramic image information inside the vehicle and the driver's foot movement image information. Based on the driving operation data and the vehicle monitoring data, it can determine in real time whether the vehicle has any abnormality. When the vehicle has an abnormality, it can brake the vehicle and automatically upload the insurance report information, thereby ensuring the user's driving safety and improving the user's driving experience.

[0147] Reference Figure 4 , an embodiment of the present invention provides a vehicle operation status monitoring system, comprising:

[0148] An image information acquisition module is used to acquire panoramic image information of the target vehicle and the driver's foot movement image information;

[0149] A driving operation recognition module is used to obtain the driver's driving operation data based on the panoramic image information and the foot movement image information;

[0150] The vehicle status judgment module is used to obtain the vehicle monitoring data of the target vehicle and determine whether the target vehicle has any abnormality based on the vehicle monitoring data and driving operation data;

[0151] The insurance reporting module is used to terminate the driving signal of the target vehicle and upload the insurance reporting information when it is determined that an abnormality has occurred in the target vehicle.

[0152] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0153] Reference Figure 5 , an embodiment of the present invention provides a vehicle running state monitoring device, comprising:

[0154] at least one processor;

[0155] at least one memory for storing at least one program;

[0156] When the at least one program is executed by the at least one processor, the at least one processor implements the vehicle operation status monitoring method.

[0157] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0158] An embodiment of the present invention further provides a computer-readable storage medium storing a program executable by a processor. When the program is executed by the processor, it is used to execute the above-mentioned vehicle operating status monitoring method.

[0159] A computer-readable storage medium according to an embodiment of the present invention can execute a vehicle operating status monitoring method provided by an embodiment of the present invention, can execute any combination of implementation steps of the embodiment of the method, and has the corresponding functions and beneficial effects of the method.

[0160] The embodiment of the present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 1 The method shown.

[0161] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0162] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skills of an engineer. Therefore, a person skilled in the art can implement the present invention set forth in the claims using ordinary skills without undue experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0163] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0164] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0165] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable media on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0166] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0167] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0168] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0169] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A vehicle operating status monitoring method, characterized in that: The following steps are involved: Obtain panoramic image information of the target vehicle interior and obtain image information of the driver's foot movements; obtaining the driver's driving operation data based on the panoramic image information and the foot movement image information; Acquiring vehicle-mounted monitoring data of the target vehicle, and determining whether an abnormality occurs with the target vehicle based on the vehicle-mounted monitoring data and the driving operation data; When it is determined that the target vehicle has an abnormality, the driving signal of the target vehicle is terminated and the insurance report information is uploaded; The step of obtaining the vehicle-mounted monitoring data of the target vehicle and determining whether an abnormality occurs in the target vehicle based on the vehicle-mounted monitoring data and the driving operation data specifically includes: Acquiring the vehicle monitoring data through a vehicle monitoring system, wherein the vehicle monitoring data includes a vehicle driving / braking state, a vehicle steering state, and a vehicle gear state; Comparing the vehicle-computer monitoring data with the driving operation data to determine whether the vehicle-computer monitoring data corresponds to the driving operation data; When the vehicle-computer monitoring data does not correspond to the driving operation data, it is determined that an abnormality occurs in the target vehicle; when the vehicle-computer monitoring data corresponds to the driving operation data, it is determined that no abnormality occurs in the target vehicle.

2. A vehicle operating status monitoring method according to claim 1, characterized in that: The step of obtaining panoramic image information of the interior of the target vehicle and obtaining image information of the driver's foot movements is specifically as follows: Acquire the panoramic image information by shooting in real time with a panoramic camera device, and acquire the foot movement image information by shooting in real time with a foot camera device; Wherein, the panoramic camera device is installed above the driver's seat of the target vehicle, and the foot camera device is installed at the bottom of the driver's seat of the target vehicle.

3. A vehicle operating status monitoring method according to claim 1, characterized in that: The step of identifying and obtaining the driver's driving operation data based on the panoramic image information and the foot movement image information specifically includes: determining hand motion image information of the driver based on the panoramic image information, determining a plurality of continuous hand motion image frames based on the hand motion image information, and further determining hand motion time series data based on the hand motion image frames; determining a plurality of continuous foot motion image frames according to the foot motion image information, and determining foot motion time series data according to the foot motion image frames; Inputting the hand motion time series data into a pre-trained hand driving operation recognition model to obtain the driver's hand driving operation data, and inputting the foot motion time series data into a pre-trained foot driving operation recognition model to obtain the driver's foot driving operation data; The driving operation data is determined based on the hand driving operation data and the foot driving operation data.

4. A vehicle operating status monitoring method according to claim 3, characterized in that: The vehicle operation status monitoring method further includes the steps of pre-training the hand driving operation recognition model and the foot driving operation recognition model, which specifically include: Obtaining a first training sample and a second training sample, wherein the first training sample is time-series data of hand movements of a test person during a test drive, and the second training sample is time-series data of foot movements of the test person during a test drive; Manually labeling the first training samples to obtain hand driving operation labels, and constructing a first training dataset based on the first training samples and the corresponding hand driving operation labels; Manually labeling the second training samples to obtain foot driving operation labels, and constructing a second training data set based on the second training samples and the corresponding foot driving operation labels; Inputting the first training data set into a pre-built first long short-term memory neural network for training, and optimizing the model parameters of the first long short-term memory neural network using a back propagation algorithm to obtain a trained hand driving operation recognition model; The second training data set is input into a pre-built second long short-term memory neural network for training, and the model parameters of the second long short-term memory neural network are optimized using a back propagation algorithm to obtain a trained foot driving operation recognition model.

5. The vehicle operation status monitoring method according to claim 1, characterized in that: The step of terminating the driving signal of the target vehicle and uploading the insurance report information when determining that the target vehicle has an abnormality is specifically as follows: When it is determined that an abnormality occurs in the target vehicle, the motor power supply of the target vehicle is cut off, and an insurance report is generated based on the vehicle monitoring data and the driving operation data, and then the insurance report is uploaded through the evidence device of the target vehicle and the user terminal of the driver.

6. A vehicle operating status monitoring method according to any one of claims 1 to 5, characterized in that: The vehicle running state monitoring method further comprises the following steps: The data exchange instantaneous flow rate and the data exchange IP address are determined by the vehicle computer monitoring data, and whether there is a hacker intrusion is determined based on the data exchange instantaneous flow rate and the data exchange IP address.

7. A vehicle operation status monitoring system, characterized in that: include: An image information acquisition module is used to acquire panoramic image information of the target vehicle and the driver's foot movement image information; a driving operation recognition module, configured to obtain the driver's driving operation data based on the panoramic image information and the foot movement image information; A vehicle status judgment module is used to obtain the vehicle monitoring data of the target vehicle and judge whether the target vehicle has any abnormality based on the vehicle monitoring data and the driving operation data; An insurance reporting module is used to terminate the driving signal of the target vehicle and upload insurance reporting information when it is determined that the target vehicle has an abnormality; The vehicle status judgment module is specifically used for: Acquiring the vehicle monitoring data through a vehicle monitoring system, wherein the vehicle monitoring data includes a vehicle driving / braking state, a vehicle steering state, and a vehicle gear state; Comparing the vehicle-computer monitoring data with the driving operation data to determine whether the vehicle-computer monitoring data corresponds to the driving operation data; When the vehicle-computer monitoring data does not correspond to the driving operation data, it is determined that an abnormality occurs in the target vehicle; when the vehicle-computer monitoring data corresponds to the driving operation data, it is determined that no abnormality occurs in the target vehicle.

8. A vehicle operation status monitoring device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a vehicle operating status monitoring method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to execute a vehicle operating status monitoring method according to any one of claims 1 to 6 when executed by the processor.

Images