Train status video recognition method, device, equipment, medium and program product

Abstract

The embodiments of the present application relate to a method, device, equipment, medium and program product for train status video recognition, and relate to the field of video information processing technology. The method comprises: performing frame-by-frame inference on each frame sequence image in the target monitoring video through a pre-trained train status recognition model to obtain the state of the vehicle in each frame image of the target monitoring video; if the number of times the same vehicle predicted state appears before the current video frame in the target monitoring video reaches a preset threshold, the vehicle predicted state is determined as the state of the vehicle in the current frame image, and a corresponding video editing is performed according to the current frame image to obtain at least one complete train status video clip corresponding to the target monitoring video. Among them, the train status recognition model is obtained by learning video temporal features based on a 2D convolutional neural network structure. The use of this method can quickly and conveniently identify a complete train video.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video information processing technology, and in particular, to a method, device, equipment, medium and program product for video recognition of train status. Background Art

[0002] With the development of video processing technology, more and more companies will deploy video acquisition equipment in target areas to monitor dynamic information in the target areas and improve the security of the company.

[0003] Taking the monitoring of vehicle information in parking areas as an example, in related technologies, a 3D neural network model is usually used to identify information such as license plate numbers, vehicle trajectories, and vehicle status in the video of the parking area, and extract the complete vehicle status video from the monitoring video based on the recognition results. However, due to the rich information in the monitoring video and the complex architecture of the 3D neural network model, which requires high terminal computing power, the related technology cannot quickly and conveniently identify the complete vehicle status video. Summary of the invention

[0004] The embodiments of the present application provide a train status video recognition method, device, equipment, medium and program product, which can be used to quickly and conveniently identify a complete train status video.

[0005] In a first aspect, an embodiment of the present application provides a method for video recognition of train status, the method comprising:

[0006] The pre-trained train status recognition model is used to infer each frame sequence image in the target monitoring video frame by frame to obtain the status of the vehicle in each frame image of the target monitoring video; the train status recognition model is trained based on the 2D convolutional neural network structure to learn the video time series features;

[0007] According to the time sequence of each frame image in the target monitoring video, for any current frame image, if the number of times the same vehicle predicted state appears before the current frame image reaches a preset threshold, the vehicle predicted state is determined as the state of the vehicle in the current frame image;

[0008] Each time the status of the vehicle in a current frame image is determined, corresponding video editing is performed according to the current frame image until corresponding video editing is completed for each frame image in the target monitoring video, thereby obtaining at least one complete vehicle status video clip corresponding to the target monitoring video.

[0009] In one embodiment, the train status recognition model includes an encoder model layer, a feature reorganization layer, and a decoder model layer. The pre-trained train status recognition model is used to perform frame-by-frame reasoning on each frame sequence image in the target monitoring video to obtain the state of the vehicle in each frame image of the target monitoring video, including:

[0010] All image features within T time in the target surveillance video are cached through the encoder model layer; T is a positive integer;

[0011] In the feature reorganization layer, all image features within time T are spliced ​​in time sequence;

[0012] The image features after time-series splicing are input into the decoder model layer to obtain the state of the vehicle in the image frame at time T, and so on, until the state of the vehicle in each frame image of the target monitoring video is obtained.

[0013] In one embodiment, all image features within T time in the target surveillance video are cached through the encoder model layer, including:

[0014] The target surveillance video is input into the encoder model layer, each frame image in the target surveillance video is downsampled in the encoder model layer, and the image features are extracted frame by frame until time T is reached, and the image features within time T are cached.

[0015] In one embodiment, the method further comprises:

[0016] Detecting image frame time information corresponding to cached image features;

[0017] According to the time information, the image features in the cache that exceed the T time interval are deleted in the order of the furthest time from the current frame.

[0018] In one embodiment, all image features within time T are subjected to time sequence stitching processing, including:

[0019] Randomly sample all image features within time T to obtain an image feature group representing a video clip;

[0020] The features in the image feature group are spliced ​​in time sequence.

[0021] In one embodiment, the image features after time sequence splicing are input into the decoder model to obtain the state of the vehicle in the image frame at time T, including:

[0022] Perform feature classification processing on the image features after time series splicing to determine the probability value of the image frame at time T belonging to different train states;

[0023] The train status with the largest probability value is determined as the status of the vehicle in the image frame at time T.

[0024] In one embodiment, each time the state of the vehicle in a current frame image is determined, a corresponding video editing is performed according to the current frame image, including at least one of the following:

[0025] When the vehicle is in a no-vehicle state, the corresponding video editing is not processed;

[0026] When the vehicle is in the vehicle driving-in state, the corresponding video is edited to start capturing the video;

[0027] The corresponding video when the vehicle is in the working state is edited to record the state time information; the working state of the vehicle at least includes the vehicle stopping state, the garbage dumping state, and the vehicle driving out state;

[0028] When the vehicle is in a state where there is no vehicle after the vehicle has driven out, the corresponding video is edited to stop capturing the video.

[0029] In one embodiment, the process of constructing the train status recognition model includes:

[0030] Obtain sample surveillance videos under a variety of different conditions; the different conditions at least include different installation positions of surveillance cameras in the surveillance area and vehicles entering and exiting the surveillance area from different directions;

[0031] Mark the vehicle status time information of each sample monitoring video;

[0032] Based on the sample surveillance videos marked with the time information of vehicle status, the initial 2D convolutional neural network model is trained to learn the temporal characteristics of each sample surveillance video until the trained 2D convolutional neural network model reaches the preset convergence condition to obtain the vehicle status recognition model.

[0033] In one embodiment, based on each sample monitoring video labeled with vehicle status time information, the initial 2D convolutional neural network model is trained to learn the temporal features of each sample monitoring video, including:

[0034] Based on the marked vehicle status time information, each sample monitoring video is framed, and each frame image obtained by frame extraction is classified into different video files according to the different vehicle status under each sample monitoring video;

[0035] Determine the training data set according to the classified video files;

[0036] Each sample surveillance video is segmented to obtain a verification data set;

[0037] The initial 2D convolutional neural network model is trained using each image frame in the training data set to learn the temporal characteristics of each sample surveillance video, and the trained 2D convolutional neural network model is verified using the verification data set.

[0038] In one embodiment, determining a training data set according to each classified video file includes:

[0039] A target image is randomly determined from each video file, and the video file where the target image is located is determined as the target video file;

[0040] Randomly select an image from the target video file as the end frame, take the time interval T before the end frame as a window, randomly select multiple images at non-uniform intervals in the window as an image group, and use the train status corresponding to the end frame as the status label of the image group;

[0041] The image group and the state label are determined as a group of training data, and the process is executed in a loop to obtain a training data set including multiple groups of training data.

[0042] In one embodiment, each sample surveillance video is segmented into segments to obtain a verification data set, including:

[0043] According to the preset time interval, each sample surveillance video is segmented to obtain multiple sub-videos;

[0044] Uniformly sample each sub-video to obtain a preset number of image groups, and use the train status corresponding to the end frame in the image group as the status label of the image group;

[0045] The image group and the state label are determined as a group of verification data, and the process is executed in a loop to obtain a verification data set including multiple groups of verification data.

[0046] In a second aspect, an embodiment of the present application provides a vehicle status video recognition device, the device comprising:

[0047] The state acquisition module is used to perform frame-by-frame reasoning on each frame sequence image in the target monitoring video through a pre-trained train state recognition model to obtain the state of the vehicle in each frame image of the target monitoring video; the train state recognition model is trained based on the 2D convolutional neural network structure to learn the video time series features;

[0048] A state judgment module is used to determine the vehicle predicted state as the state of the vehicle in the current frame image according to the time sequence of each frame image in the target monitoring video, for any current frame image, if the number of times the same vehicle predicted state appears before the current frame image reaches a preset threshold;

[0049] The video editing module is used to determine the status of the vehicle in each current frame image, perform corresponding video editing according to the current frame image, until the corresponding video editing is completed for each frame image in the target monitoring video, and obtain at least one complete vehicle status video clip corresponding to the target monitoring video.

[0050] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method in any one of the embodiments of the first aspect are implemented.

[0051] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any one of the embodiments of the first aspect above.

[0052] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method in any one of the embodiments of the first aspect above.

[0053] The train status video recognition method, device, equipment, medium and program product provided in the embodiments of the present application perform frame-by-frame reasoning on each frame sequence image in the target monitoring video through a pre-trained train status recognition model to obtain the state of the vehicle in each frame image of the target monitoring video, and according to the time sequence of each frame image in the target monitoring video, for any current frame image, if the number of times the same vehicle predicted state appears before the current frame image reaches a preset threshold, the vehicle predicted state is determined as the state of the vehicle in the current frame image; each time the state of the vehicle in a current frame image is determined, the corresponding video editing is performed according to the current frame image until the corresponding video editing is completed for each frame image in the target monitoring video, and at least one complete train status video clip corresponding to the target monitoring video is obtained. Among them, the train status recognition model is obtained by learning the video timing features based on the 2D convolutional neural network structure. It is equivalent to the structure of the train status recognition model being two-dimensional, and the train status recognition model has the function of extracting video timing features. On this basis, the trained train status recognition model is used to infer each frame sequence image in the target monitoring video, which is equivalent to using a 2D convolutional neural network structure to extract the three-dimensional information of each frame image in the target monitoring video. In this way, the information reasoning method that comprehensively considers the two-dimensional features on each frame image and the temporal features between different frame images in the target monitoring video can fully utilize the advantages of the simple structure and small number of parameters of the 2D convolutional neural network while obtaining comprehensive and rich image information, and improve the information reasoning speed, so as to quickly and accurately determine the state of the vehicle in each frame image of the target monitoring video, and then extract the complete train status video clip from the target monitoring video. In addition, due to the simple structure of the train status recognition model, the computing resources required when performing image reasoning tasks are controllable, and it can be flexibly deployed on low-computing power devices, thereby improving the scene applicability and convenience of train status video recognition. In summary, the train status video recognition method of the embodiment of the present application can be used to quickly identify the target monitoring video by deploying the train status recognition model on a low-computing power device, and obtain at least one complete train status video clip corresponding to the target monitoring video. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A diagram showing an application environment of a method for video recognition of train status in an embodiment;

[0055] Figure 2 A schematic diagram of a first process flow of a method for video recognition of train status in one embodiment;

[0056] Figure 3 A first flow chart of a vehicle status acquisition step in one embodiment;

[0057] Figure 4A schematic diagram of a flow chart of a cache image feature deletion step in one embodiment;

[0058] Figure 5 A schematic diagram of a flow chart of a cache image feature stitching step in one embodiment;

[0059] Figure 6 A second flow chart of a vehicle status acquisition step in one embodiment;

[0060] Figure 7 A first flow chart of a train status recognition model training step in one embodiment;

[0061] Figure 8 A schematic diagram of the location of monitoring equipment in a parking scenario in one embodiment;

[0062] Fig. 9 A second flow chart of a train status recognition model training step in one embodiment;

[0063] Fig.10 A schematic diagram of a flow chart of a step of obtaining a model training data set in one embodiment;

[0064] Fig.11 A schematic diagram of a flow chart of a step of obtaining a model validation data set in one embodiment;

[0065] Fig.12 A schematic diagram of the architecture of a train status recognition model in one embodiment;

[0066] Fig.13 A third flow chart of a train status recognition model training step in one embodiment;

[0067] Fig.14 A schematic diagram of a second process of a method for video recognition of train status in one embodiment;

[0068] Fig.15 It is a structural block diagram of a train status video recognition device in one embodiment;

[0069] Fig.16 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present application more clear, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the embodiments of the present application and are not used to limit the embodiments of the present application.

[0071] First of all, before specifically introducing the technical solution of the embodiments of the present application, the technical background or technical evolution context on which the embodiments of the present application are based is introduced.

[0072] With the development of video processing technology, more and more companies will deploy video acquisition equipment in target areas to facilitate information monitoring and tracing of target areas and improve the security of the company.

[0073] Taking the deployment of video acquisition equipment at the berth entrance of a garbage transfer station as an example, by identifying a complete series of vehicle states in the surveillance video, including no vehicle state, vehicle entry state, vehicle stop state, garbage dumping state, and vehicle exit state, it helps to quickly identify abnormal garbage truck driving behavior in the target area and improve the safety of the enterprise.

[0074] In actual application scenarios, in order to obtain the complete vehicle status, barriers can be installed at the entrance and exit, and the staff on the garbage truck can scan the code to enter the vehicle entry and exit information, thereby forming a complete vehicle record. However, this method requires manual participation, the steps are cumbersome, and the efficiency is low.

[0075] Furthermore, to improve the recognition efficiency of train status, operators such as 3D convolution, 3D deep convolution, and 3D point convolution can be used to build a complex neural network model to process video sequence images and identify target behaviors. However, due to the complex structure of the 3D model and the high time-consuming reasoning, the computing power of the terminal device is required to be high.

[0076] In summary, when identifying a complete train status video, the identification method in the related art either requires a lot of manpower and material resources and has low recognition efficiency, or relies on high-computing terminal equipment, which causes the recognition method to be limited by the hardware environment, making the recognition process more cumbersome. Therefore, there is an urgent need for a train status video recognition solution that can simultaneously meet the recognition speed requirements and simplify the recognition process requirements. Based on this, the embodiments of the present application provide a train status video recognition method, device, equipment, medium and program product that can quickly and conveniently identify a complete train video. Of course, the technical solution provided in the embodiments of the present application is not limited to solving only the above-mentioned problems, and there are other technical effects. Please refer to the following embodiment description for details.

[0077] Before describing the embodiments of the present application, the application scenarios of the embodiments of the present application are described first. The train status video recognition method provided by the embodiments of the present application is actually implemented inside the processor, but the program operation of the processor needs to be based on the program file, and the program file can be stored in the storage subsystem of the computer device. Therefore, the train status video recognition method provided by the embodiments of the present application can be applied to the following scenarios: Figure 1In the application environment shown, the application environment may include a terminal device 102 and a video acquisition device 104, the terminal device 102 and the video acquisition device 104 are communicatively connected, the video acquisition device 104 is used to acquire monitoring videos and transmit the monitoring videos to the terminal device 102, and the terminal device 102 identifies the train status of the received monitoring videos. The video acquisition device 104 may be a panoramic camera, a cylindrical camera, a hemispherical camera, a spherical pan / tilt camera, or other camera devices. The terminal device 102 may be a computer device, such as a personal computer, a laptop computer, a smart phone, a tablet computer, a server, and a portable wearable device, etc. The terminal device includes a processor and a storage subsystem, and the processor and the storage subsystem are communicatively connected. In another application environment, a video acquisition device may also be included, and the video acquisition device has both a video acquisition function and an identification function, or only a terminal device may be included, and the terminal device has both a video acquisition function and an identification function.

[0078] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0079] In an exemplary embodiment, a method for identifying a train status video is provided. Figure 2 As shown, the following steps are included:

[0080] S201, using a pre-trained train status recognition model to perform frame-by-frame inference on each frame sequence image in the target monitoring video, to obtain the state of the vehicle in each frame image of the target monitoring video; the train status recognition model is obtained by learning video temporal features based on a 2D convolutional neural network structure.

[0081] Among them, the train status recognition model is a deep learning model obtained by training based on a 2D convolutional neural network structure to learn video temporal features. The trained train status recognition model is a two-dimensional network structure and has the function of extracting video temporal features. In the embodiment of the present application, the input of the train status recognition model is the target monitoring video, and the output is the state of the vehicle in each frame of the target monitoring video. The state of the vehicle in the image includes but is not limited to the vehicle entering state, the vehicle working state, the vehicle exiting state, and the state in which the image does not include the vehicle. The state in which the vehicle does not exist in the image can be the state before the vehicle enters the state, or it can be the state after the vehicle exits the state.

[0082] Target surveillance video refers to the continuous video of a fixed area captured by a video capture device when the video capture device is deployed in the fixed area. For example, the garbage truck removal video collected in the parking area of ​​the garbage dump, the express delivery loading and unloading video collected at the entrance of the express station, the commodity installation video collected at the entrance of the supermarket, etc.

[0083] In practical applications, frame-by-frame reasoning of each frame sequence image in the target monitoring video through the train status recognition model means that for each frame sequence image, the static features of the vehicle in the frame sequence image are extracted through the train status recognition model, and the temporal features of the frame sequence image in the target monitoring video are extracted, and then the state of the vehicle in the frame sequence image is predicted based on the static features and the temporal features.

[0084] Optionally, the image features of multiple frames of sequence images within a preset time period are processed by time-series splicing through a train status recognition model, and the state of the vehicle in the last frame of the sequence image within the preset time period is determined based on the features after time-series splicing.

[0085] Optionally, for each frame of the sequence image, based on the timing characteristics of the frame of the sequence image, adjacent related images before / after the image are determined from the target surveillance video, and related vehicle features in each adjacent related image are extracted, and then the vehicle static features and the related vehicle features in each adjacent related image are combined to determine the state of the vehicle in the image.

[0086] Optionally, for each frame sequence image, based on the timing characteristics of the frame sequence image, the first adjacent image before the image and the second adjacent image after the image are determined from the target monitoring video, and the first associated vehicle features in each first adjacent image and the second associated vehicle features in each second adjacent image are extracted respectively, and then the first associated vehicle features, the vehicle static features, and the second associated vehicle features are combined to determine the state of the vehicle in the image.

[0087] When the state of the vehicle in each frame of the target monitoring video is determined, the complete vehicle status can be further determined and extracted from the target monitoring video.

[0088] S202, according to the time sequence of each frame image in the target monitoring video, for any current frame image, if the number of times the same vehicle predicted state appears before the current frame image reaches a preset threshold, the vehicle predicted state is determined as the state of the vehicle in the current frame image.

[0089] The vehicle prediction state of a preset number of images before the current frame image is obtained, wherein the preset number is greater than a preset threshold. In this case, if the number of times the same vehicle prediction state appears is greater than the preset threshold, the vehicle prediction state is used as the state of the vehicle in the current frame image.

[0090] Taking the preset number as N, the preset threshold as N / 2, and N as a positive integer greater than 2 as an example, if among the N vehicle prediction states corresponding to N images, the number of occurrences of the vehicle prediction state being the vehicle exit state is greater than N / 2, then the vehicle exit state is determined as the state of the vehicle in the current frame image.

[0091] In another scenario, considering the continuity of the vehicle state in the continuous frame images, the predicted vehicle state can also be determined as the state of the vehicle in the current frame image when the predicted vehicle states of the preset number of consecutive images before the current frame image are the same. For example, if the predicted vehicle states of the M consecutive frame images before the current frame image are all the vehicle exit state, then the state of the vehicle in the current frame image is determined to be the vehicle exit state. Wherein, M is a positive integer.

[0092] S203, each time the state of the vehicle in a current frame image is determined, corresponding video editing is performed according to the current frame image until the corresponding video editing is completed for each frame image in the target monitoring video, and at least one complete vehicle status video clip corresponding to the target monitoring video is obtained.

[0093] It should be noted that a complete train status includes: vehicle entry status, vehicle working status and vehicle exit status in sequence, and a complete train status video clip is a video corresponding to a complete train status of a vehicle.

[0094] In actual scenarios, the dynamic driving information of different vehicles may appear simultaneously in the target monitoring video, and the dynamic form information of the same vehicle may appear at different times. Therefore, in the embodiment of the present application, the target monitoring video at least includes a video clip of a complete vehicle status.

[0095] Taking the target monitoring video including the driving status of a vehicle as an example, the method of obtaining the video clip of the vehicle's completed train status is explained: according to the state of the vehicle in each frame image of the target monitoring video, the initial video frame of the vehicle in the vehicle driving-in state and the ending video frame of the vehicle in the vehicle driving-out state are determined respectively, and then the video between the initial video frame and the ending video frame is intercepted from the target monitoring video as a video clip of a complete train status.

[0096] In another achievable scenario, the initial video frame of the vehicle in the vehicle entry state can also be determined based on the state of the vehicle in each frame of the target monitoring video, and then the video corresponding to the preset time period after the initial video frame in the target monitoring video is obtained as a complete vehicle status video clip. The preset time period can be an average operation time determined based on the historical operation time of multiple historical vehicles, or it can be a time obtained based on the vehicle operation efficiency and working range.

[0097] The video editing includes not processing, starting to capture video, stopping to capture video, recording time information of the current frame image, etc. In the embodiment of the present application, the video editing process corresponding to a complete train status video clip is starting to capture video, recording time information, and stopping to capture video.

[0098] In actual applications, a target monitoring video includes more than one complete train status video clip, so the above video editing process can be executed multiple times. When multiple complete train status videos are not continuous in the target monitoring video, no editing is processed between two adjacent executions of the video editing process.

[0099] Taking the video editing corresponding to the vehicle entering state as the start video capture and the video editing corresponding to the vehicle exiting state as the stop video capture as an example, the Zth frame image of the target monitoring video is taken as the current frame image, and when it is determined that the state of the vehicle in the current frame image is the vehicle entering state, the editing operation of starting video capture is performed, and then the Z+1th frame image of the target monitoring video is taken as the current frame image, and when it is determined that the state of the vehicle in the current frame image is the vehicle working state, the editing operation of recording time information is performed, until the 2Zth frame of the target monitoring video is taken as the current frame image, and when it is determined that the state of the vehicle in the current frame image is the vehicle exiting state, the editing operation of stopping video capture is performed, so that the video between the Zth frame image to the 2Zth frame image is a complete vehicle status video clip.

[0100] When the number of times the same vehicle predicted state appears before the current frame image reaches a preset threshold, the vehicle predicted state is determined as the state of the vehicle in the current frame image, thereby improving the accuracy of the vehicle state prediction result. Furthermore, corresponding video editing is performed according to the state of the vehicle in the current frame image, so that the edited video clips can completely cover the vehicle entry state, vehicle working state and vehicle exit state, and will not introduce invalid videos unrelated to the vehicle state, thereby improving the integrity and effectiveness of the video clips.

[0101] In an embodiment of the present application, a pre-trained train status recognition model is used to perform frame-by-frame reasoning on each frame sequence image in the target monitoring video to obtain the state of the vehicle in each frame image of the target monitoring video, and according to the time sequence of each frame image in the target monitoring video, for any current frame image, if the number of times the same vehicle predicted state appears before the current frame image reaches a preset threshold, the vehicle predicted state is determined as the state of the vehicle in the current frame image; each time the state of the vehicle in a current frame image is determined, a corresponding video editing is performed according to the current frame image until the corresponding video editing is completed for each frame image in the target monitoring video, and at least one complete train status video clip corresponding to the target monitoring video is obtained. Among them, the train status recognition model is obtained by learning the video timing features based on a 2D convolutional neural network structure. It is equivalent to the structure of the train status recognition model being two-dimensional, and the train status recognition model has the function of extracting video timing features. On this basis, the trained train status recognition model is used to infer each frame sequence image in the target monitoring video, which is equivalent to using a 2D convolutional neural network structure to extract the three-dimensional information of each frame image in the target monitoring video. In this way, the information reasoning method that comprehensively considers the two-dimensional features on each frame image and the temporal features between different frame images in the target monitoring video can fully utilize the advantages of the simple structure and small number of parameters of the 2D convolutional neural network while obtaining comprehensive and rich image information, and improve the information reasoning speed, so as to quickly and accurately determine the state of the vehicle in each frame image of the target monitoring video, and then extract the complete train status video clip from the target monitoring video. In addition, due to the simple structure of the train status recognition model, the computing resources required when performing image reasoning tasks are controllable, and it can be flexibly deployed on low-computing power devices, thereby improving the scene applicability and convenience of train status video recognition. In summary, the train status video recognition method of the embodiment of the present application can be used to quickly identify the target monitoring video by deploying the train status recognition model on a low-computing power device, and obtain at least one complete train status video clip corresponding to the target monitoring video.

[0102] In actual scenarios, there are various ways to identify the state of a vehicle in an image using a train status recognition model, such as extracting temporal features and vehicle static features to predict the state. The following describes the architecture and recognition steps of the train status recognition model through an embodiment.

[0103] In an exemplary embodiment, Figure 3 As shown, the train status recognition model includes an encoder model layer, a feature recombination layer and a decoder model layer. The above S201 uses the pre-trained train status recognition model to perform frame-by-frame reasoning on each frame sequence image in the target monitoring video to obtain the state of the vehicle in each frame image of the target monitoring video, including the following steps:

[0104] S301, cache all image features within T time in the target surveillance video through the encoder model layer; T is a positive integer.

[0105] The image corresponding to time T in the target surveillance video is captured through the encoder model layer, and then the image features of each captured image are extracted, and the image features of the above images are summarized to obtain all image features within time T in the target surveillance video.

[0106] For example, images from time t0 to time (t0+T) in the target surveillance video are captured, and then image features of each image in the above interval are extracted and summarized to obtain all image features within time T in the target surveillance video.

[0107] S302: In the feature reorganization layer, all image features within time T are subjected to time sequence splicing processing.

[0108] According to the appearance time of the images corresponding to all image features within time T, all image features are sorted, and then all image features are spliced ​​in order from far to near in time to obtain the image features after time series splicing.

[0109] S303, input the image features after time-series splicing into the decoder model layer to obtain the state of the vehicle in the image frame at time T, and so on, until the state of the vehicle in each frame image of the target monitoring video is obtained.

[0110] The input of the decoder model layer is all the image features within time T, and the output is the state of the vehicle in the image at time T. In the embodiment of the present application, the decoder model layer is used to decode and predict the image features after the time sequence splicing process to obtain the state of the vehicle in the image at time T.

[0111] Among them, the state prediction principle of the decoder model layer is: take the image at time T as the end image, take the time interval T before the end image as the window, and predict the state of the vehicle in the end image based on the image features of all images under the window.

[0112] Steps S301 to S303 are executed in a loop. In each loop cycle, image features in a window with a time interval of T in the target surveillance video are obtained once, and the state of the vehicle in the image at the end moment of the window is predicted. The above process is continuously repeated until the state of the vehicle in each frame image of the target surveillance video is obtained.

[0113] In one embodiment, the time corresponding to the first frame image in the target monitoring video is taken as the initial time (time 0), the image features of all images in the time period [0, T] are obtained and cached through the encoder model layer, and the image features of the time period [0, T] are spliced ​​in time according to the feature reconstruction layer, and then the decoder model layer is used to decode and predict the image features after the time splicing process to obtain the state of the vehicle in the image at time T; then the time corresponding to the second frame image of the target monitoring video is taken as the new initial time (t1), the image features of all images in the time period [t1, t1+T] are obtained and cached through the encoder model layer, and according to the feature reconstruction layer, the decoder model layer is used to decode and predict the image features after the time splicing process to obtain the state of the vehicle in the image at time T; then, the time corresponding to the second frame image of the target monitoring video is taken as the new initial time (t1), the image features of all images in the time period [t1, t1+T] are obtained and cached through the encoder model layer, and according to the feature reconstruction layer, the decoder model layer is used to decode and predict the image features after the time splicing process to obtain the state of the vehicle in the image at time T; The feature reorganization layer performs time-series splicing processing on the image features of the time period [t1, t1+T], and then uses the decoder model layer to decode and predict the image features after the time-series splicing processing to obtain the state of the vehicle in the image at the moment (t1+T); then the moment corresponding to the third frame image of the target monitoring video is taken as the new initial moment (t2), and the above steps are repeated to obtain the state of the vehicle in the image at the moment (t2+T), and the above cycle is repeated. Each cycle is based on the image features of the image within the T time interval, and the state of the vehicle in the image at the moment T in each cycle period is obtained, until the state of the vehicle in each frame image of the target monitoring video is obtained.

[0114] It should be noted that for any image in the time interval from the initial moment to the moment T of the target surveillance video, since the time window before the image is less than T, it is impossible to obtain the image in the time interval T. Based on this, all the images before the image in the target surveillance video can be used as the judgment basis, and the collected images can be feature extracted, spliced ​​and predicted to obtain the state of the vehicle in the image.

[0115] In the embodiment of the present application, the encoder model layer caches the image features within T time, providing discrete and time-sequential image features for the feature reconstruction layer; the image features cached by the encoder model layer are spliced ​​in time by the feature reconstruction layer, providing the decoder model layer with continuous and time-sequential image features, so that the decoder model layer can predict the vehicle status accurately and objectively according to the time-sequential changes of the image features. Similarly, the encoder model layer, feature reconstruction layer and decoder model layer in the vehicle status recognition model cyclically perform image feature caching, splicing and status prediction, and the status of the vehicle in each frame image in the target monitoring video is also accurate and objective.

[0116] It can be seen from the above embodiments that the train status recognition model includes an encoder model layer, a feature reconstruction layer and a decoder model layer. The encoding process of the encoder model layer is further explained below.

[0117] In an exemplary embodiment, all image features within time T in the target surveillance video are cached through the encoder model layer, including:

[0118] The target surveillance video is input into the encoder model layer, each frame image in the target surveillance video is downsampled in the encoder model layer, and the image features are extracted frame by frame until time T is reached, and the image features within time T are cached.

[0119] According to the time sequence of each frame image in the target surveillance video, each frame image in the target surveillance video is downsampled in turn through the encoder model layer, and the image features of each frame image are extracted and saved until the image features of the image of the frame corresponding to time T are extracted, and the image features of all images within T time before the current time are obtained.

[0120] It should be noted that the image features cached by the encoder model layer are image features within time T, and each time the encoder model layer caches image features within time T, the feature reconstruction layer will splice the cached feature inputs and send them to the decoder model to obtain the state of the vehicle in the image corresponding to time T.

[0121] In the embodiment of the present application, time T can be regarded as a time window with a time interval of T. As the time window continues to slide, the image features cached by the encoder model layer are also dynamically changing. In addition, each time the image features cached by the encoder model layer are updated, the vehicle state prediction step is performed once.

[0122] In an embodiment of the present application, feature extraction is performed on each frame image within time T in the encoder model layer according to the time sequence of each frame image in the target monitoring video, and the consistency of the time period corresponding to the image features cached each time in the encoder model layer is maintained. The cache space requirement for the encoder model layer is relatively low, which reduces the deployment difficulty of the train status recognition model to a certain extent.

[0123] In an exemplary embodiment, Figure 4 As shown, the method further comprises the following steps:

[0124] S401, detecting image frame time information corresponding to cached image features.

[0125] In the case where each frame of the sequence image in the target monitoring video includes the monitoring time, the monitoring time of each image can be directly read from each cached image as the time information of the image.

[0126] When each frame of the sequence image in the target surveillance video does not include the surveillance time, the first frame of the target surveillance video is taken as the initial moment. For an image corresponding to any set of image features in the cache, the moment when the image appears in the entire target surveillance video is obtained as the image frame time information.

[0127] S402, according to the time information, delete the image features in the cache that are more than T time intervals in the order of the furthest time from the current frame.

[0128] The images corresponding to the image features cached in the encoder model layer are continuous in the target surveillance video, and correspondingly, the image frame time information of the cached image features is also continuous. It should be emphasized again here that the time period corresponding to the image features cached in the encoder model layer is unchanged, and they are all image features within the time period of time T.

[0129] Then, as more and more image features are cached in the encoder model layer, in order to keep the time period corresponding to the image features cached in the encoder model layer always within the T moment, the cached image features need to be deleted.

[0130] In an embodiment of the present application, the current frame cached by the encoder model layer is taken as the latest frame image, the image frame time information corresponding to the current frame is taken as the last moment within T moments, and the image features of the image within a time interval of T before the current frame are saved, and the image features of the image whose image frame time is more than T time intervals away from the current frame time are deleted.

[0131] It should be noted that the steps of caching and deleting image features are performed frame by frame, that is, each time a frame is cached, it is determined whether the maximum time interval of the currently cached image features is greater than T, and if it is greater than T, the image frame with the earliest time information in the cache space is deleted.

[0132] Taking the image frame time information originally cached by the encoder model as [0, T] as an example, if the encoder needs to continue caching the image features of the next frame of the image, it is necessary to delete the image features of the frame with the earliest time information currently cached so that the image features of the next frame can be cached. At this time, the updated cached image frame time information is [1, T+1].

[0133] In an embodiment of the present application, based on the image frame time information corresponding to the cached image features, the image features that are more than T time intervals away from the current frame time are deleted, thereby avoiding caching a large number of invalid image features in the encoder model layer and saving the cache capacity of the encoder model layer.

[0134] It can be seen from the above embodiments that the train status recognition model includes an encoder model layer, a feature reconstruction layer and a decoder model layer. The feature processing process of the feature reconstruction layer is further explained below.

[0135] like Figure 5 As shown, in an exemplary embodiment, all image features within time T are subjected to time sequence splicing processing, including the following steps:

[0136] S501, performing random sampling on all image features within time T to obtain an image feature group representing a video clip.

[0137] According to the time information of the image frame corresponding to the image feature, all the image features within the T moment are randomly sampled at unequal intervals to obtain the image features of some images within the T moment as an image feature group of a video clip.

[0138] S502, splicing the features in the image feature group in time sequence.

[0139] According to the time information of the image frames corresponding to the image features in the image feature group, the image features in the image feature group are spliced, and the image features within the T moment are fused while the order of the image features within the T moment is retained.

[0140] In an embodiment of the present application, a random number of samples are taken of all image features within time T to obtain an image feature group, and the features in the image feature group are spliced ​​in time sequence. While maintaining the time sequence of the image features within time T, the number of image features within time T is reduced to reduce the prediction difficulty of the subsequent decoder model layer and improve the state prediction efficiency.

[0141] It can be seen from the above embodiments that the train status recognition model includes an encoder model layer, a feature reconstruction layer and a decoder model layer. The decoding process of the decoder model layer is further explained below.

[0142] In an exemplary embodiment, Figure 6 As shown, the image features after time sequence splicing are input into the decoder model to obtain the state of the vehicle in the image frame at time T, including the following steps:

[0143] S601, performing feature classification processing on the image features after time sequence splicing, and determining the probability value of the image frame at time T belonging to different train states.

[0144] Taking the garbage dump parking scenario as an example, the vehicle status includes: no vehicle status, vehicle entering status, vehicle stopping status, garbage dumping status and vehicle exiting status.

[0145] In this case, the image features after time series splicing are predicted to obtain a first probability value that the vehicle is in a no-vehicle state, a second probability value that the vehicle is in a vehicle-entering state, a third probability value that the vehicle is in a stopped state, a fourth probability value that the vehicle is in a garbage dumping state, and a fifth probability value that the vehicle is in a vehicle-exiting state in the image frame at time T.

[0146] S602: Determine the vehicle status with the largest probability value as the status of the vehicle in the image frame at time T.

[0147] The probability values ​​of the image frame at time T belonging to different train states are compared to determine the train state corresponding to the maximum probability value, that is, the state of the vehicle in the image frame at time T.

[0148] In an embodiment of the present application, feature classification processing is performed on the image features after time series splicing, and the possibility of the image frame at time T belonging to different train states is quantitatively evaluated with probability values, and the train state with the largest probability value is determined as the state of the vehicle in the image frame at time T. The classification basis is reliable, the determination process is quantitative and intuitive, and the accuracy of the state determination result is improved.

[0149] Taking the application of the vehicle status video recognition method in the scenario of monitoring the driving status of garbage trucks in the parking area of ​​a garbage dump as an example, the corresponding relationship between the vehicle status and video editing is explained.

[0150] In an exemplary embodiment, each time the state of the vehicle in a current frame image is determined, a corresponding video editing is performed according to the current frame image, including at least one of the following:

[0151] When the vehicle is in a no-vehicle state, the corresponding video editing is not processed;

[0152] When the vehicle is in the vehicle driving-in state, the corresponding video is edited to start capturing the video;

[0153] The corresponding video when the vehicle is in the working state is edited to record the state time information; the working state of the vehicle at least includes the vehicle stopping state, the garbage dumping state, and the vehicle driving out state;

[0154] When the vehicle is in a state where there is no vehicle after the vehicle has driven out, the corresponding video is edited to stop capturing the video.

[0155] In the scenario of extracting a complete video clip of the status of a garbage truck, if the status of the vehicle in the current frame image in the target monitoring video is a no-vehicle status, no video editing is performed; then the next frame image of the current frame is used as the new current frame image, and the vehicle status of the new current frame image is obtained, until the status of the vehicle in the current frame image is a vehicle-entering status, and video capture begins; then as the current frame image is continuously updated, the status of the vehicle in the current frame image is also continuously updated, and the time information corresponding to the vehicle status update is recorded, including: the time information corresponding to switching from the vehicle-entering status to the vehicle-stopping status, that is, the time when the vehicle starts to dump garbage, and the time information corresponding to switching from the vehicle-dumping status to the vehicle-exiting status, that is, the time when the vehicle ends dumping garbage; finally, the status of the vehicle in the current frame image is updated to a no-vehicle status after the vehicle has driven out, and video capture stops, so that a video clip of a complete trip status including vehicle-entering, vehicle-stopping, garbage-dumping, and vehicle-exiting is obtained.

[0156] In an embodiment of the present application, the status of the vehicle in the image is subdivided into five categories: no vehicle state, entry state, vehicle stop state, garbage dumping state and vehicle exit state, and a corresponding video editing operation is set for each state, so as to quickly and accurately determine the complete vehicle status video clip from the target monitoring video, thereby improving the efficiency of vehicle status video recognition and extraction.

[0157] The above embodiment describes the application process of the train status recognition model in the train status recognition method, that is, identifying the state of the vehicle in the image. Next, the construction process of the train status recognition model is described. In an exemplary embodiment, Figure 7 As shown in FIG. 1 , the construction process of the train status recognition model includes the following steps:

[0158] S701, obtaining sample surveillance videos under a variety of different conditions; the different conditions at least include different installation positions of surveillance cameras in the surveillance area and vehicles entering and exiting the surveillance area from different directions.

[0159] From the surveillance cameras deployed at different installation locations in the surveillance area, sample surveillance videos collected by each surveillance camera are obtained.

[0160] For example, the monitoring area is the parking area of ​​the garbage dump, and the sample monitoring video is the video collected by the camera installed in the parking area of ​​the garbage dump. Figure 8 As shown, Figure 8 This is a schematic diagram of the installation location of surveillance cameras in the parking area of ​​the garbage dump. Figure 8 It can be seen that corresponding surveillance cameras can be installed in different parking spaces, so that different surveillance cameras can monitor vehicles entering and leaving from different directions from different angles, and obtain multiple sample surveillance videos. Figure 8 The surveillance camera at berth 1 can monitor sample surveillance videos of vehicles entering berth 1, vehicles working in berth 1, and vehicles leaving berth 1; the surveillance camera at berth 2 can monitor sample surveillance videos of vehicles entering berth 2, vehicles working in berth 2, and vehicles leaving berth 2; the surveillance camera at berth 3 can monitor sample surveillance videos of vehicles entering berth 3, vehicles working in berth 3, and vehicles leaving berth 3.

[0161] S702: Mark the vehicle status time information of each sample monitoring video.

[0162] For each sample surveillance video, the actual image acquisition time and the status of the vehicle in the image are marked on each frame image in the sample surveillance video. In other words, each sample surveillance video in the embodiment of the present application includes multiple frames of images, and the vehicle status time information of each frame image includes a vehicle status label and a time information label.

[0163] S703: Based on each sample monitoring video labeled with vehicle status time information, the initial 2D convolutional neural network model is trained to learn the temporal characteristics of each sample monitoring video until the trained 2D convolutional neural network model reaches a preset convergence condition to obtain a vehicle status recognition model.

[0164] The sample surveillance videos annotated with the vehicle status time information are input into the initial 2D convolutional neural network model. The initial 2D convolutional neural network model is used to learn the temporal characteristics of each sample surveillance video. The vehicle status time information annotated in each sample surveillance video is used as the prediction target, and logical reasoning training is performed until the preset convergence conditions are met. The training is completed and the trained 2D convolutional neural network model is determined as the train status recognition model.

[0165] In one scenario, the preset convergence condition can be that the model reaches a preset number of iterations, and each sample surveillance video is input into the initial 2D convolutional neural network model in batches for training until all sample surveillance videos are input once as a round of training. The above steps are repeated until the number of model training rounds reaches the preset number of iterations, and a trained 2D convolutional neural network model is obtained, that is, the train status recognition model.

[0166] In an embodiment of the present application, sample surveillance videos of surveillance cameras at different installation positions in the surveillance area and vehicles entering and exiting the surveillance area from different directions are obtained, and the vehicle status time information is annotated to enrich the diversity of data set samples and labels, avoid overfitting of the initial 2D convolutional neural network model, and improve the training efficiency of the 2D convolutional neural network model.

[0167] In an exemplary embodiment, Fig. 9As shown, the above S703 trains the initial 2D convolutional neural network model to learn the temporal characteristics of each sample monitoring video based on each sample monitoring video annotated with the vehicle status time information, including the following steps:

[0168] S901, extract frames from each sample monitoring video based on the marked vehicle status time information, and classify each frame image obtained by the extraction into different video files according to different vehicle statuses under each sample monitoring video.

[0169] For any sample surveillance video, each frame of the sample surveillance video is discretized to obtain an image set corresponding to the sample surveillance video, and then multiple images corresponding to each sample surveillance video are obtained.

[0170] Furthermore, for each image in each sample surveillance video, the image is divided into video files of the same train status according to the train status corresponding to the image, until each image in all sample surveillance videos is divided, and video files of different train status are obtained. In other words, each type of video file includes multiple images of one train status.

[0171] S902: Determine a training data set according to the classified video files.

[0172] The video file is an image set including multiple images, the training data set and the verification data set can be multiple image sets, and the number of images in different image sets can be the same or different.

[0173] Optionally, for any type of video file, a portion of images are randomly divided from the video file as training images, and the training images are combined in different ways to obtain multiple groups of training data, thereby obtaining a training data set.

[0174] S903, segment each sample surveillance video into segments to obtain a verification data set.

[0175] For each sample surveillance video, the sample surveillance video is first segmented according to a preset time interval to obtain multiple video clips, and then a preset number of images are randomly selected from each video clip as a set of verification data.

[0176] For each sample surveillance video, the sample surveillance video is first segmented according to a preset time interval to obtain multiple video clips, and then a preset number of images are randomly selected from each video clip as a set of verification data.

[0177] S904, using each image frame in the training data set to train the initial 2D convolutional neural network model to learn the temporal characteristics of each sample surveillance video, and using the verification data set to verify the trained 2D convolutional neural network model.

[0178] A training data set including vehicle state annotation information is input into an initial 2D convolutional neural network model. The initial 2D convolutional neural network model uses the temporal features between image frames in the training data set as a training basis and the vehicle state of each image frame in the training data set as a training target to perform logical reasoning training to obtain a trained 2D convolutional neural network model. Then, a verification data set that does not include vehicle state annotation information is input into the trained 2D convolutional neural network model to obtain a predicted vehicle state of the verification data set. Then, based on the predicted vehicle state and the vehicle state annotation of the verification data set, the prediction error of the trained 2D convolutional neural network model is obtained to realize verification of the 2D convolutional neural network model.

[0179] Furthermore, if the prediction error is less than or equal to the preset error threshold, it is determined that the training of the 2D convolutional neural network model is completed. If the prediction error is greater than the preset error threshold, it is determined that the training of the 2D convolutional neural network model is not completed. At this time, you can return to step S902, re-divide the training data set and the verification data set, and perform cross-training, and repeat S902~S903 until the prediction error is less than or equal to the preset error threshold.

[0180] In an embodiment of the present application, based on the labeled vehicle status time information, each sample surveillance video is subjected to image discretization, image classification, combination and other processing to obtain a training data set and a verification data set with timing characteristics, and then the initial 2D convolutional neural network is trained according to the training data set and the verification data set, so that the trained 2D convolutional neural network has the ability to learn timing characteristics, supports the extraction of timing features in the target surveillance video, and improves the processing efficiency of the target surveillance video.

[0181] The following is an example of how to obtain a training data set. Fig.10 As shown, according to the classified video files, the training data set is determined, including:

[0182] S1001, randomly determine a target image from each video file, and determine the video file where the target image is located as the target video file.

[0183] The images in each video file are regarded as a full image set, and then an image is randomly selected from the full image set as the target image, and the video file to which the target image belongs is regarded as the target video file, so as to select a set of training data from the target video file later.

[0184] S1002, randomly select an image from the target video file as the end frame, take the time interval T before the end frame as a window, randomly select multiple images at non-uniform intervals in the window as a group of images, and use the train status corresponding to the end frame as the status label of the image group.

[0185] An image is randomly selected from the target video file as the end frame again, and (N-1) images are randomly selected within T time before the end frame in the target video file, and then the (N-1) images and the end frame are combined into an image group including N images as a set of training data.

[0186] It should be noted that although the end frame and the target image are both determined from the same target video file, their functions are completely different. In the embodiment of the present application, the target image is determined from the full set of images for the purpose of determining the video file where the training data is located, and the end frame is determined from the video file for the purpose of determining the content of the training data.

[0187] S1003, determining the image group and the state label as a group of training data, and executing the process in a loop to obtain a training data set including multiple groups of training data.

[0188] Each set of training data corresponds to a category of video files, and the state label of each set of training data is the category of the video file to which the images in the set of training data belong.

[0189] In an embodiment of the present application, a target image is randomly determined from each video file, and the target video file to which the target image belongs is used as the data source of a set of training data sets. On this basis, according to a fixed time interval, an end frame and multiple images before the end frame are randomly determined from the target video file to form a group of image groups corresponding to the time interval, and training data is obtained, which is equivalent to reducing multiple images in the time interval into a smaller number of image groups, so as to reduce the amount of parameters that need to be learned for model training, thereby improving the model training speed.

[0190] The following is an example of an example to illustrate the steps of obtaining a verification data set. Fig.11 As shown, each of the sample surveillance videos is segmented to obtain a verification data set, including:

[0191] S1101, dividing each sample surveillance video into multiple sub-videos according to a preset time interval.

[0192] Taking the time interval T as an example, each sample surveillance video is segmented to obtain multiple sub-videos with a duration of T.

[0193] S1102, uniformly sample each sub-video to obtain a preset number of image groups, and use the train status corresponding to the end frame in the image group as a status label of the image group.

[0194] For each segmented sub-video, multiple images are uniformly collected in the sub-video to obtain a preset number of image groups as a set of verification data. Since the length of each sub-video is the same, the number of images in the image group collected by each sub-video is also the same.

[0195] Each set of verification data includes multiple images, and the vehicle state corresponding to the end frame in the multiple images can be determined as a label of the verification data.

[0196] In another scenario, the vehicle state that appears most frequently in multiple images can also be determined as the label of the verification data.

[0197] S1103, determining the image group and the state label as a group of verification data, and executing in a loop to obtain a verification data set including multiple groups of verification data.

[0198] In an embodiment of the present application, each sample surveillance video is segmented at a preset time interval to ensure the consistency of the time length of each sub-video, and then sampled evenly from each sub-video to obtain an image group that can comprehensively represent the status of all vehicles in the sub-video as verification data. This is equivalent to randomly cutting out multiple groups of verification data from the surveillance video. This method of obtaining a verification data set is completely different from the method of determining a training data set based on a video file. The greater the difference between the verification data set and the training data set, the more objective and accurate the verification result accuracy of the vehicle status recognition model will be.

[0199] In one embodiment, a vehicle status video recognition method is provided, which is applied to a vehicle monitoring scenario in a garbage dump parking area.

[0200] In this scenario, the vehicle status includes five types: no vehicle status, vehicle entry status, vehicle stop status, garbage dump status, and vehicle exit status. In addition, the vehicle status recognition model for identifying the vehicle status can be a residual network model (ResNet), such as Fig.12 As shown, Fig.12 The figure is a schematic diagram of the architecture of the ResNet model. The ResNet model includes an encoder, a feature recombination layer and a decoder. The encoder includes an input layer, a convolutional layer and a residual module, which is used to extract and cache the image features of each frame image within time T in the target monitoring video. The feature recombination layer is used to splice the image features within time T cached by the encoder according to the timing features. The decoder includes a residual module and a linear change module, which are used to predict the state of the features spliced ​​by the feature recombination layer and output the state of the vehicle in the image at time T.

[0201] like Fig.13 As shown, Fig.13 This is a schematic diagram of the reasoning steps of the above residual network model, including:

[0202] S1301, receiving input target surveillance video.

[0203] S1302: Cache all image features of the target surveillance video within time T through an encoder.

[0204] The input dimension of the encoder is [B*N, 3, H, W], where B represents the number of picture groups, each of which represents a video clip; N represents the number of pictures contained in the picture group; 3 represents the RGB channels of the image; H and W represent the height and width of the input image.

[0205] After the encoder downsamples the image in the video, the output dimension of the encoder is [B*N, C, H / 4, W / 4], where C represents the number of feature channels. It should be noted that the height and width of the image in the output dimension depends on the magnitude of the encoder downsampling, and can also be H / 8, W / 8, etc., without limitation.

[0206] S1303, randomly sampling image features of N frames of images from all images within time T through a feature recombination layer, and splicing them together.

[0207] The input dimension of the feature reconstruction layer is the same as the output dimension of the encoder. The output dimension of the feature reconstruction layer is [B, N*C, H / 4, W / 4], which means that the feature reconstruction layer concatenates the features of N pictures of a video clip cached by the encoder.

[0208] S1304, predicting the image features spliced ​​by the feature reorganization layer through a decoder.

[0209] S1305, output the probability that the vehicle state in the last frame image at time T belongs to different vehicle states.

[0210] When the vehicle states include no vehicle state, vehicle entering state, vehicle stopping state, garbage dumping state, and vehicle exiting state, the dimension of the output result can be expressed as [B, 5], which represents the probability that group B of pictures belongs to the above five types of vehicle states.

[0211] In the embodiment of the present application, the features of multiple image frames in different time periods are cached by the encoder, so that each image frame only needs to be encoded once, avoiding the need to extract encoding features of the image frame each time thereafter, thereby avoiding waste of encoding resources. Furthermore, the newly added reorganization layer splices the multiple continuous image frames obtained by the encoder, which is equivalent to taking into account the temporal feature coherence of the current image frame and the historical image frame, so that the train status result determined in this way is more accurate.

[0212] like Fig.14 As shown, Fig.14 The flowchart of the train status video recognition method in the above scenario includes the following steps:

[0213] S1401, input the target surveillance video to the encoder, downsample each frame of the target surveillance video in the encoder, and extract image features frame by frame until reaching time T, and cache the image features within time T.

[0214] In practical applications, the image frame time information corresponding to the cached image features in the cache is detected, and according to the time information, the image features in the cache that exceed the T time interval are deleted in the order of the furthest distance from the current frame time.

[0215] S1402: Randomly sample image features of N images for all image features within time T to obtain an image feature group representing a video clip.

[0216] S1403, performing feature classification processing on the image features after time sequence splicing to determine the probability value of the image frame at time T belonging to different train statuses.

[0217] S1404: Determine the vehicle status with the largest probability value as the status of the vehicle in the image frame at time T.

[0218] S1405, according to the time sequence of each frame image, for any current frame image, if the number of times the same vehicle predicted state appears before the current frame image reaches a preset threshold, the vehicle predicted state is determined as the state of the vehicle in the current frame image.

[0219] S1406: If the vehicle is in a no-vehicle state, video editing is not performed.

[0220] S1407, if the vehicle is in the vehicle driving-in state, then start capturing the video.

[0221] S1408, if the vehicle is in a working state, continue to infer and predict the vehicle's stopping state, garbage dumping state, and vehicle exiting state, and record the state time information.

[0222] S1409: If the vehicle is in a state of no vehicle after the vehicle has driven out and the number of times reaches a threshold, then stop capturing video clips and generate a complete train video clip.

[0223] S1410, performing corresponding video editing on each frame image in the target monitoring video to obtain at least one complete train status video segment corresponding to the target monitoring video.

[0224] In the embodiment of the present application, during the training process, in view of the changes in viewing angles and video images caused by different installation positions of the surveillance camera, video data under various complex conditions are collected for model training and fitting, so as to enrich the training data and realize adaptive intelligent adaptation of the scene. In practical applications, a 2D convolutional neural network model with simple structure and efficient calculation is used to identify the different vehicle states in the video, so as to conveniently obtain the time points of each state, with superior processing speed, taking into account both reasoning performance and recognition accuracy, and having good scalability.

[0225] It should be understood that, although the various steps in the above flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0226] In one embodiment, Fig.15 As shown, a train status video recognition device is provided, including: a status acquisition module 1501, a status judgment module 1502 and a video editing module 1503, wherein:

[0227] The state acquisition module 1501 is used to perform frame-by-frame reasoning on each frame sequence image in the target monitoring video through a pre-trained train state recognition model to obtain the state of the vehicle in each frame image of the target monitoring video; the train state recognition model is obtained by learning the video time series feature training based on a 2D convolutional neural network structure;

[0228] The state judgment module 1502 is used to determine the vehicle predicted state as the state of the vehicle in the current frame image according to the time sequence of each frame image in the target monitoring video, for any current frame image, if the number of times the same vehicle predicted state appears before the current frame image reaches a preset threshold;

[0229] The video editing module 1503 is used to determine the status of the vehicle in a current frame image each time, and perform corresponding video editing according to the current frame image until the corresponding video editing is completed for each frame image in the target monitoring video, thereby obtaining at least one complete vehicle status video clip corresponding to the target monitoring video.

[0230] In one embodiment, the state acquisition module 1501 includes: a feature cache unit, a feature splicing unit and a feature decoding unit, wherein:

[0231] A feature cache unit is used to cache all image features within T time in the target surveillance video through the encoder model layer; T is a positive integer;

[0232] A feature splicing unit, used for performing temporal splicing processing on all image features within time T in the feature reorganization layer;

[0233] The feature decoding unit is used to input the image features after time-series splicing into the decoder model layer to obtain the state of the vehicle in the image frame at time T, and so on, until the state of the vehicle in each frame image of the target monitoring video is obtained.

[0234] In one embodiment, the feature cache unit is also used to input the target surveillance video into the encoder model layer, downsample each frame image in the target surveillance video in the encoder model layer, and extract image features frame by frame until reaching time T, and cache the image features within time T.

[0235] In one embodiment, the feature cache unit is further used to detect image frame time information corresponding to cached image features; based on the time information, image features in the cache that exceed T time intervals are deleted in the order of the furthest distance from the current frame time.

[0236] In one embodiment, the feature splicing unit includes: a feature sampling subunit and a feature timing splicing subunit, wherein:

[0237] The feature sampling subunit is used to randomly sample all image features within time T to obtain an image feature group representing a video clip;

[0238] The feature time sequence splicing subunit is used to splice the features in the image feature group in time sequence.

[0239] In one embodiment, the feature decoding unit includes: a probability acquisition subunit and a probability value determination subunit, wherein:

[0240] The probability acquisition subunit is used to perform feature classification processing on the image features after time sequence splicing, and determine the probability value of the image frame at time T belonging to different train states;

[0241] The probability value judgment subunit is used to determine the vehicle status with the largest probability value as the status of the vehicle in the image frame at time T.

[0242] In one embodiment, the video editing module 1503 includes at least one of the following:

[0243] When the vehicle is in a no-vehicle state, the corresponding video editing is not processed;

[0244] When the vehicle is in the vehicle driving-in state, the corresponding video is edited to start capturing the video;

[0245] The corresponding video when the vehicle is in the working state is edited to record the state time information; the working state of the vehicle at least includes the vehicle stopping state, the garbage dumping state, and the vehicle driving out state;

[0246] When the vehicle is in a state where there is no vehicle after the vehicle has driven out, the corresponding video is edited to stop capturing the video.

[0247] In one embodiment, the above-mentioned train status video recognition device further includes: a video acquisition module, a video annotation module and a model training module, wherein:

[0248] A video acquisition module is used to acquire sample surveillance videos under various conditions; the different conditions at least include different installation positions of surveillance cameras in the surveillance area and vehicles entering and exiting the surveillance area from different directions;

[0249] The video annotation module is used to annotate the vehicle status time information of each sample monitoring video;

[0250] The model training module is used to train the initial 2D convolutional neural network model to learn the temporal characteristics of each sample monitoring video based on the sample monitoring videos marked with the vehicle status time information, until the trained 2D convolutional neural network model reaches the preset convergence condition, thereby obtaining the vehicle status recognition model.

[0251] In one embodiment, the above-mentioned model training module includes: an image classification unit, a data set division unit and a data set training unit, wherein:

[0252] An image classification unit is used to extract frames from each sample monitoring video based on the marked vehicle status time information, and classify each frame image obtained by the extraction into different video files according to the different vehicle status under each sample monitoring video;

[0253] A training data set determination unit, used to determine a training data set according to each classified video file;

[0254] A verification data set determination unit, used to segment each of the sample surveillance videos into segments to obtain a verification data set;

[0255] The data set training unit is used to use each image frame in the training data set to train the initial 2D convolutional neural network model to learn the temporal characteristics of each sample surveillance video, and to use the verification data set to verify the trained 2D convolutional neural network model.

[0256] In one embodiment, the training data set determination unit further includes: an image screening subunit, an end frame determination subunit and a training data set acquisition subunit, wherein:

[0257] The image screening subunit is used to randomly determine a target image from each video file, and determine the video file where the target image is located as the target video file;

[0258] The end frame determination subunit is used to randomly select an image from the target video file as the end frame, take the time interval T before the end frame as a window, randomly select multiple images at non-uniform intervals in the window as a group of images, and use the train status corresponding to the end frame as the status label of the image group;

[0259] The training data set acquisition subunit is used to determine the image group and the state label as a group of training data, and execute in a loop to obtain a training data set including multiple groups of training data.

[0260] In one embodiment, the verification data set determination unit further includes: a video slicing subunit, an image sampling subunit and a verification data set acquisition subunit, wherein:

[0261] The video segmentation sub-unit is used to segment each sample surveillance video according to a preset time interval to obtain multiple sub-videos;

[0262] The image sampling subunit is used to uniformly sample in each sub-video to obtain a preset number of image groups, and use the train status corresponding to the end frame in the image group as the status label of the image group;

[0263] The verification data set acquisition subunit is used to determine the image group and the state label as a group of verification data, and execute in a loop to obtain a verification data set including multiple groups of verification data.

[0264] For the specific definition of the train status video recognition device, please refer to the definition of the train status video recognition method above, which will not be repeated here. Each module in the above-mentioned train status video recognition device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0265] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.16 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for video recognition of train status is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0266] Those skilled in the art will understand that Fig.16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0267] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0268] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0269] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0270] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0271] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the embodiments of the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0272] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0273] The above-described embodiments only express several implementation methods of the embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the embodiments of the present application, and these all belong to the protection scope of the embodiments of the present application. Therefore, the protection scope of the patent of the embodiments of the present application shall be subject to the attached claims.

Claims

1. A method for train status video recognition, characterized in that: The method comprises: The vehicle static features in each frame sequence image in the target monitoring video and the temporal features of each frame sequence image in the target monitoring video are extracted through a pre-trained train status recognition model, and the status of the vehicle in each frame image of the target monitoring video is obtained based on the static features and the temporal features; the train status recognition model is obtained by learning the video temporal features based on a 2D convolutional neural network structure; According to the time sequence of each frame image in the target monitoring video, for any current frame image, if the number of times the same vehicle predicted state appears before the current frame image reaches a preset threshold, the vehicle predicted state is determined as the state of the vehicle in the current frame image; Each time the status of the vehicle in a current frame image is determined, corresponding video editing is performed according to the current frame image until the corresponding video editing is completed for each frame image in the target monitoring video, and at least one complete vehicle status video clip corresponding to the target monitoring video is obtained; the video editing includes no processing, starting video capture, stopping video capture, and recording the time information of the current frame image.

2. The method according to claim 1, characterized in that The train status recognition model includes an encoder model layer, a feature reorganization layer and a decoder model layer. The train status recognition model trained in advance extracts static features of vehicles in each frame sequence image in the target monitoring video, as well as temporal features of each frame sequence image in the target monitoring video, and obtains the state of the vehicle in each frame image of the target monitoring video according to the static features and the temporal features, including: Cache all image features within T time in the target surveillance video through the encoder model layer; T is a positive integer; In the feature reorganization layer, all image features within the T time are subjected to time sequence splicing processing; The image features after time-series splicing are input into the decoder model layer to obtain the state of the vehicle in the image frame at time T, and so on, until the state of the vehicle in each frame image of the target monitoring video is obtained.

3. The method according to claim 2, characterized in that The caching of all image features within time T in the target surveillance video through the encoder model layer includes: The target surveillance video is input to the encoder model layer, each frame image in the target surveillance video is downsampled in the encoder model layer, and image features are extracted frame by frame until time T is reached, and the image features within time T are cached.

4. The method according to claim 3, characterized in that The method further comprises: Detecting image frame time information corresponding to cached image features; According to the time information, image features in the cache that exceed a time interval of T are deleted in the order of the furthest time from the current frame.

5. The method according to any one of claims 2 to 4, characterized in that: The performing time-series splicing processing on all image features within the time T includes: Randomly sampling all image features within the time T to obtain an image feature group representing a video clip; The features in the image feature group are spliced ​​in time sequence.

6. The method according to any one of claims 2 to 4, characterized in that: The step of inputting the image features after time sequence splicing into the decoder model to obtain the state of the vehicle in the image frame at time T includes: Performing feature classification processing on the image features after the time sequence splicing to determine the probability value of the image frame at time T belonging to different train states; The train status with the largest probability value is determined as the status of the vehicle in the image frame at time T.

7. The method according to any one of claims 1 to 4, characterized in that: The step of determining the state of the vehicle in a current frame image and performing corresponding video editing according to the current frame image includes at least one of the following: When the vehicle is in a no-vehicle state, the corresponding video editing is not processed; When the vehicle is in the vehicle driving-in state, the corresponding video is edited to start capturing the video; The corresponding video when the vehicle is in the working state is edited to record the state time information; the working state of the vehicle at least includes the vehicle stopping state, the garbage dumping state, and the vehicle driving out state; When the vehicle is in a state where there is no vehicle after the vehicle has driven out, the corresponding video is edited to stop capturing the video.

8. The method according to any one of claims 1 to 4, characterized in that: The construction process of the train status recognition model includes: Obtain sample surveillance videos under a variety of different conditions; the different conditions at least include different installation positions of surveillance cameras in the surveillance area and vehicles entering and exiting the surveillance area from different directions; Marking the vehicle status time information of each of the sample monitoring videos; Based on each of the sample monitoring videos marked with vehicle status time information, the initial 2D convolutional neural network model is trained to learn the temporal characteristics of each of the sample monitoring videos until the trained 2D convolutional neural network model reaches a preset convergence condition to obtain the vehicle status recognition model.

9. The method according to claim 8, characterized in that The initial 2D convolutional neural network model is trained to learn the temporal characteristics of each of the sample monitoring videos based on each of the sample monitoring videos annotated with the vehicle status time information, including: Extract frames from each of the sample monitoring videos based on the marked vehicle status time information, and classify each frame image obtained by the extraction into different video files according to different vehicle statuses under each of the sample monitoring videos; Determine the training data set according to the classified video files; Segment each of the sample surveillance videos into segments to obtain a verification data set; The initial 2D convolutional neural network model is trained using each image frame in the training data set to learn the temporal characteristics of each sample surveillance video, and the trained 2D convolutional neural network model is verified using the verification data set.

10. The method according to claim 9, characterized in that Determining the training data set according to the classified video files includes: Randomly determine a target image from each of the video files, and determine the video file where the target image is located as the target video file; Randomly select an image from the target video file as the end frame, take the time interval T before the end frame as a window, randomly select multiple images at unequal intervals in the window as a group of images, and use the train status corresponding to the end frame as the status label of the image group; The image group and the state label are determined as a group of training data, and the process is executed in a loop to obtain a training data set including multiple groups of training data.

11. The method according to claim 9, characterized in that The segmentation of each sample surveillance video to obtain a verification data set includes: According to a preset time interval, each of the sample surveillance videos is segmented to obtain a plurality of sub-videos; uniformly sampling in each of the sub-videos to obtain a preset number of image groups, and using the train status corresponding to the end frame in the image group as the status label of the image group; The image group and the state label are determined as a group of verification data, and the process is executed in a loop to obtain a verification data set including multiple groups of verification data.

12. A train status video recognition device, characterized in that: The device comprises: A state acquisition module is used to extract static features of vehicles in each frame sequence image in a target monitoring video and temporal features of each frame sequence image in the target monitoring video through a pre-trained train state recognition model, and obtain the state of the vehicle in each frame image of the target monitoring video based on the static features and temporal features; the train state recognition model is obtained by learning video temporal features based on a 2D convolutional neural network structure; A state judgment module is used to determine the vehicle predicted state as the state of the vehicle in the current frame image for any current frame image according to the time sequence of each frame image in the target monitoring video, if the number of times the same vehicle predicted state appears before the current frame image reaches a preset threshold; The video editing module is used to determine the status of the vehicle in a current frame image each time, and perform corresponding video editing according to the current frame image until the corresponding video editing is completed for each frame image in the target monitoring video, thereby obtaining at least one complete vehicle status video clip corresponding to the target monitoring video; the video editing includes not processing, starting to capture video, stopping to capture video, and recording the time information of the current frame image.

13. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the vehicle status video recognition method described in any one of claims 1 to 11 when executing the computer program.

14. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, is used to implement the steps of the vehicle status video recognition method according to any one of claims 1 to 11.

15. A computer program product, characterized in that When the computer program is executed by a processor, the steps of the vehicle status video recognition method described in any one of claims 1 to 11 are implemented.

Images