Non-motor vehicle state estimation method and device, storage medium, and electronic device

Abstract

The present invention provides a method and device for estimating the state of a non-motor vehicle, a storage medium, and an electronic device, wherein the method comprises obtaining non-motor vehicle image information in a video by target tracking; inputting multiple consecutive frames of the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle, wherein the preset learning model is a convolutional neural network model obtained by training multiple sets of sample image data, and the multiple sets of sample image data include image data of multiple consecutive frames of the non-motor vehicle in the image information; and determining the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle. Through the present invention, the problem of improving the accuracy of non-motor vehicle state recognition is solved, thereby achieving the effect of improving recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation, and in particular to a method and device for estimating the state of a non-motor vehicle, a storage medium, and an electronic device. Background Art

[0002] The use of artificial semi-intelligent methods in urban road traffic management can no longer meet current requirements. As the traffic volume continues to increase and the scenarios become more numerous and wider, non-motor vehicle intelligent control systems are usually used to estimate the status of non-motor vehicles.

[0003] However, the estimation results are inefficient and require manual elimination of erroneous samples, which affects subsequent violation judgments and reduces the efficiency of the traffic management department.

[0004] There is currently no effective solution to the problem of improving the accuracy of non-motor vehicle status recognition in related technologies. Summary of the invention

[0005] Embodiments of the present invention provide a method and device for estimating the state of a non-motor vehicle, a storage medium, and an electronic device, so as to at least solve the problem of improving the accuracy of identifying the state of a non-motor vehicle in the related art.

[0006] According to one embodiment of the present invention, a method for estimating the state of a non-motor vehicle is provided, comprising: acquiring non-motor vehicle image information in a video through target tracking; inputting multiple consecutive frames of the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle, wherein the preset learning model is a convolutional neural network model obtained by training multiple groups of sample image data, and the multiple groups of sample image data include multiple consecutive frames of image data of the non-motor vehicle in the image information; and determining the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle.

[0007] According to another embodiment of the present invention, a non-motor vehicle state estimation device is provided, comprising: an acquisition module, used to acquire non-motor vehicle image information in a video by target tracking; an estimation module, used to input multiple consecutive frames of the non-motor vehicle image information into a preset learning model to obtain a state estimation result of the non-motor vehicle, wherein the preset learning model is a convolutional neural network model obtained by training multiple groups of sample image data, and the multiple groups of sample image data include image information containing multiple consecutive frames of image data of the non-motor vehicle; and a determination module, used to determine the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle.

[0008] According to yet another embodiment of the present invention, a storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0009] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0010] Through the present invention, since the non-motor vehicle image information in the video is obtained by target tracking, and multiple consecutive frames of the non-motor vehicle image information are input into a preset learning model to obtain the state estimation result of the non-motor vehicle, the problem of improving the accuracy of non-motor vehicle state recognition can be solved, and the effect of greatly improving the recognition accuracy can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0012] Figure 1 It is a hardware structure block diagram of a mobile terminal of a method for estimating a state of a non-motor vehicle according to an embodiment of the present invention;

[0013] Figure 2 is a flow chart of a method for estimating a state of a non-motor vehicle according to an embodiment of the present invention;

[0014] Figure 3 is a structural block diagram of a non-motor vehicle state estimation device according to an embodiment of the present invention;

[0015] Figure 4 is a flow chart of a method for estimating a state of a non-motor vehicle according to an optional embodiment of the present invention;

[0016] Figure 5 is a flow chart of Bi-GRU-Attention recognition in a state estimation method for a non-motor vehicle according to an optional embodiment of the present invention;

[0017] Figure 6 Schematic diagram of the Bi-GRU-Attention model structure in a non-motor vehicle state estimation method according to an optional embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0020] Example 1

[0021] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a non-motor vehicle state estimation method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0022] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the state estimation method of the non-motor vehicle in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0023] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0024] In this embodiment, a method for estimating the state of a non-motor vehicle running on the above-mentioned mobile terminal is provided. Figure 2 is a flow chart of a method for estimating a state of a non-motor vehicle according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0025] Step S202, obtaining non-motor vehicle image information in the video by target tracking;

[0026] Step S204, inputting multiple consecutive frames of the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle, wherein the preset learning model is a convolutional neural network model obtained by training multiple sets of sample image data, and the multiple sets of sample image data include multiple consecutive frames of image data of the non-motor vehicle in the image information;

[0027] Step S206 , determining the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle.

[0028] Through the above steps, since the non-motor vehicle image information in the video is obtained by target tracking, and the continuous multiple frames of the non-motor vehicle image information are input into the preset learning model to obtain the state estimation result of the non-motor vehicle, the problem of improving the accuracy of non-motor vehicle state recognition can be solved, and the effect of greatly improving the recognition accuracy can be achieved.

[0029] After performing target detection and target tracking processing on the video in the above step S202, the non-motor vehicle image information in the video can be obtained.

[0030] It should be noted that the above-mentioned target detection and target tracking processing methods may include multiple methods, which are not specifically limited in the embodiments of the present invention, and those skilled in the art may select them according to actual conditions.

[0031] In the above step S204, a plurality of continuous image frames in the image information of the non-motor vehicle are input into the preset learning model, and the state estimation result of the non-motor vehicle is output through the learning model.

[0032] The preset learning model adopts a convolutional neural network model, and the preset learning model is obtained through training. When training the model, the image information containing multiple consecutive frames of the non-motor vehicle can be used as multiple groups of image sample data to train the model.

[0033] In the above step S206, the current state of the non-motor vehicle can be determined according to the state result of the non-motor vehicle, so as to obtain a more accurate state estimation result.

[0034] As an optional embodiment of the present invention, the step of inputting the image information of the non-motor vehicle into a preset learning model according to multiple consecutive frames to obtain a state estimation result of the non-motor vehicle includes: obtaining a target frame in the image information of the non-motor vehicle whose current state needs to be determined; and inputting a sequence of image information of multiple consecutive frames before the target frame into the preset learning model to obtain a state estimation result of the non-motor vehicle.

[0035] In specific implementation, multiple frames can be input into the preset learning model according to 5 consecutive frames. The 5 consecutive frames are determined by obtaining the target frame whose current state needs to be determined in the image information of the non-motor vehicle, and the image information sequence of the 5 consecutive frames before the target frame is input into the preset learning model, and the state estimation result of the non-motor vehicle is output.

[0036] As an optional embodiment of the present invention, the method of inputting the image information of the non-motor vehicle into a preset learning model according to continuous frames to obtain a state estimation result of the non-motor vehicle includes: using continuous frames in the image information of the non-motor vehicle as input features of the convolutional neural network model; the input features are sequentially passed through a coding layer, a gated recurrent connection layer, and an attention network layer to obtain a state estimation result of the non-motor vehicle, wherein the coding layer is used to encode the input features, the gated recurrent connection layer is used to perform association learning on an output result of the coding layer, and the attention network layer is used to perform attention learning on an output result of the gated recurrent connection layer.

[0037] In a specific implementation, the network layer results of the preset learning model include a coding layer, a gated recurrent connection layer, and an attention network layer. According to the feature input in the convolutional neural network, the coding layer is sequentially encoded, the gated recurrent connection layer is used for association learning, and the attention network layer is used for attention learning.

[0038] As an optional embodiment of the present invention, determining the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle includes: when the state estimation result of the non-motor vehicle is an upward state or a downward state of the non-motor vehicle, and the target detection result of the last frame in the image information sequence of multiple consecutive frames includes the non-motor vehicle and the human body, determining that the current state of the non-motor vehicle is an upward state or a downward state; when the state estimation result of the non-motor vehicle is a state in which the non-motor vehicle is pushed by a person, and the target detection result of the last frame in the image information sequence of multiple consecutive frames includes the non-motor vehicle and the human body, determining that the current state of the non-motor vehicle is a state in which the non-motor vehicle is pushed by a person.

[0039] During specific implementation, the state estimation result of the non-motor vehicle obtained according to the preset learning model may be an up / down state, that is, the up / down state is output. At the same time, the target detection result of the last frame of the target must include a bicycle and a human body. If so, the state of the non-motor vehicle is determined to be an up / down state, otherwise the output is an unknown state.

[0040] According to the preset learning model, the state estimation result of the non-motor vehicle may be a human pushing a cart, that is, 5 consecutive frames of pictures are input into the network, and the output is whether it is a human pushing a cart. If yes, the target detection result of the last frame of the target must contain a bicycle and a human body, otherwise the output is an unknown state.

[0041] As an optional embodiment of the present invention, determining the current state of the non-motor vehicle based on the state estimation result of the non-motor vehicle includes: when the state estimation result of the non-motor vehicle is a rollover and unmanned state of the non-motor vehicle, and the target detection results in the image information sequence of multiple consecutive frames include the non-motor vehicle and a human body, determining the current state of the non-motor vehicle is a rollover and unmanned state of the non-motor vehicle; when the state estimation result of the non-motor vehicle is a rollover and manned state of the non-motor vehicle, and the target detection results in the image information sequence of multiple consecutive frames include the non-motor vehicle and / or a human body falls to the ground, determining the current state of the non-motor vehicle is a rollover and manned state of the non-motor vehicle.

[0042] In specific implementation, the state estimation result of the non-motor vehicle obtained according to the preset learning model may be a rollover of the vehicle without a driver. That is, 5 consecutive frames of pictures are input into the network, and the output is whether the vehicle without a driver is in a rollover state. If yes, if there is a non-motor vehicle or a human target detected in one frame of the target detection results of the 5 frames of pictures, if not, the output is no.

[0043] According to the preset learning model, the state estimation result of the non-motor vehicle may be that the vehicle and the person are rolling over at the same time. That is, 5 consecutive frames of images are input into the network, and the output is whether the vehicle and the person are rolling over at the same time. If so, if there is a frame of the target detection results of the 5 frames of images that detects the non-motor vehicle falling to the ground or the pedestrian falling to the ground, the output is that the vehicle and the person are rolling over at the same time. If so, the state is set to no.

[0044] As an optional embodiment of the present invention, determining the current state of the non-motor vehicle based on the state estimation result of the non-motor vehicle includes: when the state estimation result of the non-motor vehicle is the state of the non-motor vehicle runover light, determining that the current state of the non-motor vehicle is the state of the non-motor vehicle runover light; when the state estimation result of the non-motor vehicle is the collision state of the non-motor vehicle, and the target detection results in the image information sequence of multiple consecutive frames include non-motor vehicles, pedestrians or motor vehicles, determining that the current state of the non-motor vehicle is the collision state of the non-motor vehicle.

[0045] In specific implementation, the estimated state result of the non-motor vehicle obtained according to the preset learning model may be a non-motor vehicle runover light, that is, 5 consecutive frames of pictures are input into the network, and the output is whether it is a non-motor vehicle runover light state.

[0046] According to the preset learning model, the estimated state of the non-motor vehicle may be a non-motor vehicle collision. That is, 5 consecutive frames of images are input into the network, and the output is whether it is a non-motor vehicle collision state. If it is, if there is no other non-motor vehicle, pedestrian, or motor vehicle target detected in any frame of the target detection results of the 5 frames of images. If not, the output is no.

[0047] As an optional embodiment of the present invention, the method of obtaining non-motor vehicle image information in a video through target tracking includes: marking the position of the target object in the video and a target detection frame according to a preset target detection method; and obtaining non-motor vehicle image information in the video according to the target object according to the target detection frame.

[0048] In the specific implementation, target detection is performed on the video, and the coordinate positions of motor vehicles, non-motor vehicles, pedestrians, human bodies, human heads, and head and shoulders in each frame of the video are given. Target tracking is performed based on the target detection results, and an independent ID is set for each target based on the target detection frame. Pedestrians, non-motor vehicles, and head and shoulders are used as the main tracking targets, the human body is bound to the non-motor vehicle tracking target, and the face, head, and human body are bound to the head and shoulders target as auxiliary tracking targets.

[0049] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0050] Example 2

[0051] In this embodiment, a non-motor vehicle state estimation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0052] Figure 3 is a structural block diagram of a non-motor vehicle state estimation device according to an embodiment of the present invention. Figure 3 As shown, the device includes

[0053] An acquisition module 30, used for acquiring non-motor vehicle image information in the video through target tracking;

[0054] The estimation module 32 is used to input multiple consecutive frames of the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle, wherein the preset learning model is a convolutional neural network model obtained by training multiple sets of sample image data, and the multiple sets of sample image data include multiple consecutive frames of image data of the non-motor vehicle in the image information;

[0055] The determination module 34 is used to determine the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle.

[0056] After the acquisition module 30 performs target detection and target tracking processing on the video, the non-motor vehicle image information in the video can be obtained.

[0057] It should be noted that the above-mentioned target detection and target tracking processing methods may include multiple methods, which are not specifically limited in the embodiments of the present invention, and those skilled in the art may select them according to actual conditions.

[0058] After the estimation module 32 performs target detection and target tracking processing on the video, the non-motor vehicle image information in the video can be obtained.

[0059] It should be noted that the above-mentioned target detection and target tracking processing methods may include multiple methods, which are not specifically limited in the embodiments of the present invention, and those skilled in the art may select them according to actual conditions.

[0060] The determination module 34 can determine the current state of the non-motor vehicle according to the state result of the non-motor vehicle, and obtain a more accurate state estimation result.

[0061] The estimation module 32 is used to obtain a target frame whose current state needs to be determined in the image information of the non-motor vehicle; input a sequence of image information of a plurality of consecutive frames before the target frame into the preset learning model to obtain a state estimation result of the non-motor vehicle.

[0062] The estimation module 32 is used to use the continuous frames in the image information of the non-motor vehicle as the input features of the convolutional neural network model; the input features are sequentially passed through the encoding layer, the gated recurrent connection layer, and the attention network layer to obtain the state estimation result of the non-motor vehicle, wherein the encoding layer is used to encode the input features, the gated recurrent connection layer is used to perform association learning on the output results of the encoding layer, and the attention network layer is used to perform attention learning on the output results of the gated recurrent connection layer.

[0063] The determination module 34 is used to determine that the current state of the non-motor vehicle is an upward state or a downward state when the state estimation result of the non-motor vehicle is an upward state or a downward state of the non-motor vehicle and the target detection result of the last frame in the image information sequence of multiple consecutive frames includes the non-motor vehicle and the human body; and to determine that the current state of the non-motor vehicle is a state in which the non-motor vehicle is pushed by a person and the target detection result of the last frame in the image information sequence of multiple consecutive frames includes the non-motor vehicle and the human body.

[0064] The determination module 34 is used to determine that the current state of the non-motor vehicle is a state in which the non-motor vehicle has rolled over and is unmanned, when the state estimation result of the non-motor vehicle is a state in which the non-motor vehicle has rolled over and is unmanned, and the target detection results in the image information sequence of multiple consecutive frames include the non-motor vehicle and a human body; and to determine that the current state of the non-motor vehicle is a state in which the non-motor vehicle has rolled over and is manned, when the state estimation result of the non-motor vehicle is a state in which the non-motor vehicle has rolled over and is manned, and the target detection results in the image information sequence of multiple consecutive frames include the non-motor vehicle and / or a human body has fallen.

[0065] The determination module 34 is used to determine that the current state of the non-motor vehicle is the state of the non-motor vehicle runover light when the state estimation result of the non-motor vehicle is the state of the non-motor vehicle runover light; and to determine that the current state of the non-motor vehicle is the state of the non-motor vehicle collision when the state estimation result of the non-motor vehicle is the state of the non-motor vehicle collision and the target detection results in the image information sequence of multiple consecutive frames include non-motor vehicles, pedestrians or motor vehicles.

[0066] The acquisition module 30 is used to mark the location of the target object and the target detection frame in the video according to a preset target detection method; and acquire the non-motor vehicle image information in the video according to the target detection frame and the target object.

[0067] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0068] In order to better understand the process of the above-mentioned non-motor vehicle state estimation method, the above-mentioned technical solution is explained below in combination with a preferred embodiment, but it is not used to limit the technical solution of the embodiment of the present invention.

[0069] The preferred embodiment of the present invention utilizes the time series information of a multi-frame image sequence to simultaneously identify multiple states of a non-motor vehicle through a convolutional neural network model, and combines target detection and tracking strategies to accurately estimate the driving state and posture of the non-motor vehicle.

[0070] like Figure 4 , which is a flow chart of a method for estimating the state of a non-motor vehicle in an optional embodiment of the present invention.

[0071] like Figure 5 , which is a Bi-GRU-Attention recognition flow chart in a state estimation method for a non-motor vehicle in an optional embodiment of the present invention.

[0072] like Figure 6 , which is a schematic diagram of the Bi-GRU-Attention model structure in the state estimation method of a non-motor vehicle in an optional embodiment of the present invention.

[0073] exist Figure 4In the front-end camera, target detection / target tracking module, non-motor vehicle posture estimation module and human-vehicle association module are included. When performing target detection, the yolov3 algorithm is used to give the coordinate positions of motor vehicles, non-motor vehicles, pedestrians, human bodies, human heads, and head-shoulders in each frame of the video. When performing target tracking, the siamrpn++ algorithm is used to set an independent ID for each target according to the target detection frame (that is, a new ID is assigned from the first time the target appears in the video until the target disappears from the video for 10 seconds and the ID cache is cancelled). Pedestrians, non-motor vehicles and head-shoulders are taken as the main tracking targets, the human body is bound to the non-motor vehicle tracking target, and the face, head and human body are bound to the head-shoulder target as auxiliary tracking targets.

[0074] The non-motor vehicle posture estimation module performs non-motor vehicle posture estimation. Specifically, the present invention adopts the Bi-GRU-Attention model, and for 25 frames of video, the non-motor vehicle target detection frame of 5 consecutive frames (one frame is taken every two frames) is expanded to twice the original frame and sent to the network, such as Figure 5 As shown, the current state of the target is finally output through multiple classifiers. It is a model with multiple frames corresponding to multiple outputs, including up / down, people pushing the car, vehicle rollover without a driver, simultaneous rollover of people and vehicles, non-motor vehicle crushing lights, and non-motor vehicle collision states.

[0075] For up / down state: output the up / down state, and the target detection result of the last frame of the target must contain bicycles and human bodies, otherwise the output is unknown state;

[0076] For the state of a person pushing a cart: 5 consecutive frames of images are input into the network, and the output is whether it is a state of a person pushing a cart. At the same time, the target detection result of the last frame of the target must contain a bicycle and a human body, otherwise the output is an unknown state;

[0077] For the rollover state of a vehicle without a driver: 5 consecutive frames of images are input into the network, and the output is whether the vehicle is in a rollover state without a driver. At the same time, if there is a frame of the target detection results of the 5 frames of images that detects a non-motor vehicle-human target, the output is no;

[0078] For the state of simultaneous rollover of a person and a vehicle: input 5 consecutive frames of images into the network, and output whether it is a state of simultaneous rollover of a person and a vehicle. At the same time, if there is a frame of the target detection results of the 5 frames of images in which a non-motor vehicle or a pedestrian falls to the ground (the two states can exist in different frames), then output that the person and the vehicle have rolled over at the same time, otherwise the state is set to no;

[0079] For the status of non-motor vehicle runover lights: 5 consecutive frames of images are input into the network, and the output is whether it is the status of non-motor vehicle runover lights;

[0080] For the non-motor vehicle collision status: 5 consecutive frames of pictures are input into the network, and the output is whether it is a non-motor vehicle collision status. At the same time, if no frame of the target detection results of the 5 frames of pictures detects other non-motor vehicles, pedestrians, or motor vehicle targets, the output is no.

[0081] like Figure 6 As shown in the structural diagram of the Bi-GRU-Attention model proposed in the optional embodiment of the present invention, cnn_feature_input interface Figure 5 The cnn unit in the embedding layer is used to encode the cnn_feature_input. The bidirectional GRU gated recurrent unit connection layer can learn the association between the input features and the previous and next contexts. Figure 5 Input five pictures of CNN features directly related, this layer can be multi-layer, in this patent we use a double-layer bidirectional GRU connection layer, and then use the attention layer to learn the attention of each GRU output, which can find the important features of the results of the GRU before and after the association, while suppressing the unimportant features, and finally send the output results to Figure 6 In the softmax of , the recognition result is output.

[0082] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0083] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0084] S1, obtaining non-motor vehicle image information in the video through target tracking;

[0085] S2, inputting multiple consecutive frames of the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle, wherein the preset learning model is a convolutional neural network model obtained by training multiple sets of sample image data, and the multiple sets of sample image data include multiple consecutive frames of image data of the non-motor vehicle in the image information;

[0086] S3, determining the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle.

[0087] Optionally, the storage medium is further configured to store a computer program for performing the following steps:

[0088] S1, obtaining a target frame whose current state needs to be determined in the image information of the non-motor vehicle;

[0089] S2, inputting a sequence of image information of a plurality of consecutive frames before the target frame into the preset learning model to obtain a state estimation result of the non-motor vehicle.

[0090] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0091] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0092] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0093] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0094] S1, obtaining non-motor vehicle image information in the video through target tracking;

[0095] S2, inputting multiple consecutive frames of the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle, wherein the preset learning model is a convolutional neural network model obtained by training multiple sets of sample image data, and the multiple sets of sample image data include multiple consecutive frames of image data of the non-motor vehicle in the image information;

[0096] S3, determining the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle.

[0097] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0098] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for estimating the state of a non-motor vehicle, characterized in that: include: Obtain non-motor vehicle image information in the video through target tracking; Inputting multiple consecutive frames of the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle, wherein the preset learning model is a convolutional neural network model obtained by training multiple sets of sample image data, and the multiple sets of sample image data include multiple consecutive frames of image data containing the non-motor vehicle in the image information; Determining the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle; The step of inputting a plurality of consecutive frames in the image information of the non-motor vehicle into a preset learning model to obtain the state estimation result of the non-motor vehicle comprises: using the consecutive frames in the image information of the non-motor vehicle as input features of the convolutional neural network model; the input features sequentially pass through a coding layer, a gated recurrent connection layer, and an attention network layer to obtain the state estimation result of the non-motor vehicle, wherein the coding layer is used to encode the input features, the gated recurrent connection layer is used to perform association learning on the output results of the coding layer, and the attention network layer is used to perform attention learning on the output results of the gated recurrent connection layer; The method of inputting multiple consecutive frames of the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle includes: enlarging the target detection frame included in the multiple consecutive frames by two times, inputting each enlarged detection frame into multiple classifiers included in the preset learning model, and determining the state estimation result based on the states output by the multiple classifiers, wherein multiple frames correspond to multiple classifiers.

2. The method according to claim 1, characterized in that: The step of inputting a plurality of consecutive frames of the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle comprises: Acquire a target frame whose current state needs to be determined in the image information of the non-motor vehicle; A sequence of image information of a plurality of consecutive frames before the target frame is input into the preset learning model to obtain a state estimation result of the non-motor vehicle.

3. The method according to claim 2, characterized in that Determining the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle includes: When the state estimation result of the non-motor vehicle is an upward state or a downward state of the non-motor vehicle, and the target detection result of the last frame in the image information sequence of multiple consecutive frames includes the non-motor vehicle and the human body, determining that the current state of the non-motor vehicle is an upward state or a downward state; When the state estimation result of the non-motor vehicle is a state where the non-motor vehicle is pushed by a person, and the target detection result of the last frame in a sequence of image information of multiple consecutive frames includes a non-motor vehicle and a human body, it is determined that the current state of the non-motor vehicle is a state where the non-motor vehicle is pushed by a person.

4. The method according to claim 2, characterized in that: Determining the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle includes: When the state estimation result of the non-motor vehicle is that the non-motor vehicle is overturned and unmanned, and the target detection results in the image information sequence of multiple consecutive frames include the non-motor vehicle and the human body, determining that the current state of the non-motor vehicle is that the non-motor vehicle is overturned and unmanned; When the state estimation result of the non-motor vehicle is that the non-motor vehicle has rolled over and is driven by someone, and the target detection results in the image information sequence of multiple consecutive frames include non-motor vehicles, and / or a human body has fallen to the ground, it is determined that the current state of the non-motor vehicle is that the non-motor vehicle has rolled over and is driven by someone.

5. The method according to claim 2, characterized in that: Determining the current state of the non-motor vehicle according to the state estimation result of the non-motor vehicle includes: In a case where the state estimation result of the non-motor vehicle is the state of the non-motor vehicle runover light, determining that the current state of the non-motor vehicle is the state of the non-motor vehicle runover light; When the state estimation result of the non-motor vehicle is a collision state of the non-motor vehicle, and the target detection results in the image information sequence of multiple consecutive frames include non-motor vehicles, pedestrians or motor vehicles, it is determined that the current state of the non-motor vehicle is a collision state of the non-motor vehicle.

6. The method according to claim 1, characterized in that The obtaining of non-motor vehicle image information in the video by target tracking includes: Marking the location of the target object and the target detection frame in the video according to a preset target detection method; According to the target detection frame and the target object, non-motor vehicle image information in the video is obtained.

7. A non-motor vehicle state estimation device, characterized in that: include: An acquisition module, used for acquiring non-motor vehicle image information in the video through target tracking; an estimation module, used for inputting a plurality of consecutive frames in the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle, wherein the preset learning model is a convolutional neural network model obtained by training a plurality of sets of sample image data, and the plurality of sets of sample image data include a plurality of consecutive frames of image data containing the non-motor vehicle in the image information; A determination module, configured to determine a current state of the non-motor vehicle according to a state estimation result of the non-motor vehicle; The estimation module implements the step of inputting the continuous multiple frames in the image information of the non-motor vehicle into the preset learning model to obtain the state estimation result of the non-motor vehicle in the following manner: using the continuous frames in the image information of the non-motor vehicle as the input features of the convolutional neural network model; the input features are sequentially passed through the encoding layer, the gated recurrent connection layer, and the attention network layer to obtain the state estimation result of the non-motor vehicle, wherein the encoding layer is used to encode the input features, the gated recurrent connection layer is used to perform association learning on the output results of the encoding layer, and the attention network layer is used to perform attention learning on the output results of the gated recurrent connection layer; The estimation module implements the step of inputting multiple consecutive frames of the image information of the non-motor vehicle into a preset learning model to obtain a state estimation result of the non-motor vehicle in the following manner: enlarging the target detection frame included in the multiple consecutive frames by two times, inputting each enlarged detection frame into multiple classifiers included in the preset learning model, and determining the state estimation result based on the states output by the multiple classifiers, where multiple frames correspond to multiple classifiers.

8. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Images