Methods, apparatus and electronic devices for storing video

Abstract

This application relates to a method, apparatus, and electronic device for storing video. The method includes: obtaining video data, including environmental video data of a mobile platform; processing the video data using a trained accident prediction model to obtain an accident prediction result; and if the accident prediction result indicates that an accident has occurred on the mobile platform, writing video data for a preset time period prior to the current moment into a non-volatile memory. This application can effectively reduce the storage space occupied by useless video data in non-volatile memory and improve the storage security and reliability of useful video data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and electronic device for storing video. Background Technology

[0002] With the rapid development of computer technology and artificial intelligence technology, artificial intelligence technology is being applied to more and more scenarios, such as intelligent transportation and image recognition.

[0003] To preserve evidence for pursuing accountability or achieving exemption from liability in scenarios such as traffic accidents, in-vehicle devices can be used to record and store evidence while the vehicle is in motion. However, the storage space of in-vehicle devices is usually limited, which may result in useful video data not being stored or being overwritten by useless video data. For example, dashcams of this technology store video data recorded during vehicle movement on a memory card with a fixed size. If the memory card contains a large amount of useless video data, occupying a significant amount of storage space, it may prevent useful video data from being recorded, or the useful video data recorded earlier may be overwritten by the useless video data recorded later. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a method, apparatus, and electronic device for storing video, which can effectively reduce the storage space occupied by useless video data and improve the storage security and reliability of useful video data.

[0005] The first aspect of this application provides a method for storing video, comprising:

[0006] Acquire video data, including environmental video data from the mobile platform;

[0007] Video data is processed using a trained accident prediction model to obtain accident prediction results;

[0008] If the accident prediction result indicates that an accident has occurred on the mobile platform, then the video data for a preset time period prior to the current moment will be written into non-volatile memory (NVM).

[0009] According to certain embodiments of this application, the accident prediction model includes:

[0010] The feature acquisition module is used to extract feature maps from video frames in the video data;

[0011] The candidate box acquisition module is used to generate K×M candidate boxes based on M pixels of the feature map, and select N candidate boxes that include the target object from the K×M candidate boxes, where K is the preset number of candidate boxes for a pixel, and N and M are integers greater than 1.

[0012] The candidate feature acquisition module is used to obtain candidate features corresponding to each of the N candidate boxes based on the feature map.

[0013] The recognition module is used to determine the accident prediction result of the candidate box corresponding to the candidate feature based on the candidate feature.

[0014] According to certain embodiments of this application, the candidate box obtaining module is specifically used to adjust at least one of the size parameters of K×M candidate boxes, and select N candidate boxes from the adjusted K×M candidate boxes as N detection boxes;

[0015] The candidate feature acquisition module is specifically used to obtain the detection features corresponding to each of the N detection boxes based on the feature map;

[0016] The recognition module is specifically used to determine the accident prediction result based on the detection features and the corresponding detection box.

[0017] According to certain embodiments of this application, the feature acquisition module includes:

[0018] Multiple convolutional pairs, and pooling layers disposed between partially adjacent convolutional pairs, wherein each convolutional pair includes an adjacent convolutional layer and an activation layer.

[0019] According to certain embodiments of this application, the candidate box obtaining module includes:

[0020] The first branch is used to classify the pixels of the feature map based on the activation function to obtain the category of the candidate box. The category includes: including the target object and not including the target object. The number of pixels in the feature map is related to the number of pooling layers.

[0021] The second branch is used to calculate the offset relative to the candidate box;

[0022] The candidate box determination layer is used to determine candidate boxes, including the target object, and the coverage of the candidate boxes on the feature map based on the candidate box category and the candidate box offset.

[0023] According to certain embodiments of this application, the second branch includes:

[0024] The adjustment module is used to adjust at least one of the multiple adjustment dimensions of any one of the K candidate boxes corresponding to each pixel in the feature map until the candidate box covers the target object and the coverage area of ​​the candidate box is minimized. The multiple adjustment dimensions of the candidate box include at least one of the following: top, bottom, left and right.

[0025] The offset determination module is used to determine the offset of the candidate box corresponding to the pixel based on the average of the adjusted offsets of the K candidate boxes.

[0026] According to certain embodiments of this application, the identification module is specifically used to determine the accident prediction result of the candidate box corresponding to the candidate feature based on the candidate feature and the road segment attribute feature. The road segment attribute feature includes at least one of the following: highway grade, accident-prone road segment, school road segment or residential area road segment.

[0027] According to certain embodiments of this application, the accident prediction model is trained in the following manner:

[0028] The basic model consisting of a feature acquisition module, a candidate feature acquisition module, and a recognition module is trained based on the training data to obtain the trained basic model. The candidate boxes required by the candidate feature acquisition module can be obtained by processing the training data with the model including the trained candidate box acquisition module.

[0029] The feature acquisition module, candidate box acquisition module, candidate feature acquisition module, and recognition module of the trained base model are sequentially spliced ​​together to obtain the accident prediction model to be trained.

[0030] The training data is input into the feature acquisition module, and the model parameters of the candidate box acquisition module are adjusted so that the output of the accident prediction model to be trained is close to the label data corresponding to the training data.

[0031] According to certain embodiments of this application, the training data includes: videos collected by the mobile platform and corresponding tags for the videos, wherein the tags include at least one of the following: pedestrian lying down, pedestrian interfering with the mobile platform, interference between mobile platforms, pedestrian waving, pedestrian in running posture, and pedestrian in walking posture.

[0032] According to certain embodiments of this application, the above method further includes:

[0033] If the accident prediction result indicates that no accident has occurred on the mobile platform, then a video stream for the preset time period prior to the current moment is generated, but this video stream is not stored; or

[0034] If the accident prediction result indicates that no accident has occurred on the mobile platform, then the video data after the current moment will overwrite the video data within the preset time period before the current moment.

[0035] A second aspect of this application provides an apparatus for storing video, comprising: a video data acquisition module, a video data processing module, and a video data storage module.

[0036] The video data acquisition module is used to acquire video data, which includes environmental video data of the mobile platform.

[0037] The video data processing module is used to process video data using a trained accident prediction model, determine and output accident prediction results;

[0038] The video data storage module is used to write video data for a preset time period prior to the current moment into a non-volatile memory if the accident prediction result indicates that an accident has occurred on the mobile platform.

[0039] A third aspect of this application provides an electronic device, including: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method described above.

[0040] A fourth aspect of this application also provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the above-described method.

[0041] The fifth aspect of this application also provides a computer program product including executable code that, when executed by a processor, implements the above-described method.

[0042] The method, apparatus, and electronic device for storing video provided in this application analyze video data to predict whether there is a significant risk of a traffic accident occurring on the mobile platform. Only if the prediction indicates a significant risk of a traffic accident will the video data be stored in non-volatile memory, rather than storing all video data. This embodiment effectively increases the proportion of useful video data in the stored video data and reduces the storage space occupied by useless video data. This saves storage space, improves the reliability of timely storage of useful video data, and reduces the possibility of useful video data being overwritten by useless video data.

[0043] In some embodiments, candidate feature data for classification is selected from the feature map based on candidate bounding boxes, effectively reducing the number of feature data to be classified. This reduces computational load while increasing focus on useful feature data, contributing to improved recognition performance and speed.

[0044] In some embodiments, multiple candidate boxes are generated for each pixel in the extracted feature map, and each candidate box is adjusted to obtain the offset of multiple candidate boxes for that pixel. By taking the average value, the accuracy of the determined candidate boxes can be effectively improved.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0046] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0047] Figure 1 An exemplary system architecture, applicable to methods, apparatus, and electronic devices for storing video, is shown according to embodiments of this application.

[0048] Figure 2 This illustration schematically depicts an application scenario of storing video according to an embodiment of this application;

[0049] Figure 3 A flowchart illustrating a method for storing video according to an embodiment of this application is shown schematically;

[0050] Figure 4 This illustration schematically shows a structural diagram of an accident prediction model according to an embodiment of this application;

[0051] Figure 5 This illustration schematically shows a structural diagram of a feature extraction module according to an embodiment of this application;

[0052] Figure 6 A schematic diagram illustrating a series of candidate boxes according to embodiments of this application is shown;

[0053] Figure 7 This illustration schematically shows a structural diagram of a candidate box obtaining module according to an embodiment of the present application;

[0054] Figure 8 The illustration shows a schematic diagram of the candidate feature acquisition process according to an embodiment of this application;

[0055] Figure 9 This illustration schematically shows a flowchart of a training accident prediction model according to an embodiment of the present application;

[0056] Figure 10 A block diagram of an apparatus for storing video according to an embodiment of this application is shown schematically;

[0057] Figure 11 A block diagram of an electronic device according to an embodiment of this application is shown schematically. Detailed Implementation

[0058] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0059] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The terms "comprising," "including," etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0060] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0061] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0062] The dashcam technology described above records and stores video data taken while the vehicle is in motion on a memory card with a fixed storage capacity. This results in the storage of video footage regardless of whether a traffic accident has occurred, leading to an accumulation of junk data (such as useless video data) on the memory card. The applicant discovered that out of several hours (or tens of hours) of video data, only a few minutes (e.g., about 10 minutes) might be useful.

[0063] Furthermore, as dashcams are used for extended periods, users are required to manually delete stored video data to free up memory card space, which is extremely inconvenient. If memory card space isn't freed up in time, in the event of a traffic accident, useful video data may not be recorded due to the delayed release of storage space, or it may be overwritten by later-recorded, useless video data, leading to irreparable losses.

[0064] This application's embodiments utilize an accident prediction model to identify objects such as vehicles and pedestrians in a video stream, predicting the likelihood of an accident. For example, if the probability of a collision between a vehicle and a pedestrian is greater than a preset probability threshold, it is determined that an accident is imminent, and recording permissions for non-volatile memory are granted to store useful video data in the non-volatile memory. Otherwise, a video stream is generated without actual storage, or a video stream overwrite operation is performed. This reduces the storage of useless video data while ensuring the preservation of useful video data.

[0065] The following will be through Figures 1 to 10 This application provides a detailed description of a method, apparatus, and electronic device for storing video according to embodiments of the present application.

[0066] Figure 1 An exemplary system architecture, applicable to methods, apparatus, and electronic devices for storing video, is illustrated according to embodiments of this application. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this application, in order to help those skilled in the art understand the technical content of this application, but do not mean that the embodiments of this application cannot be used in other devices, systems, environments or scenarios.

[0067] See Figure 1 The system architecture 100 according to this embodiment may include mobile platforms 101, 102, and 103, a network 104, and a cloud 105. The network 104 serves as a medium for providing communication links between the mobile platforms 101, 102, and 103 and the cloud 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. Mobile terminals, such as dashcams, may be mounted on the mobile platforms 101, 102, and 103 to perform functions such as recording and storing videos.

[0068] Users can use mobile platforms 101, 102, and 103 to interact with other mobile platforms and the cloud 105 via network 104 to receive or send information, such as sending model training requests, model parameter download requests, and receiving trained model parameters. Mobile platforms 101, 102, and 103 can have various communication client applications installed, such as driver assistance applications, autonomous driving applications, in-vehicle applications, web browser applications, database applications, search applications, instant messaging tools, email clients, social media platform software, and so on.

[0069] Mobile platforms 101, 102, and 103 include, but are not limited to, electronic devices such as cars, robots, tablets, and laptops that can support functions such as internet access, video recording, and human-computer interaction.

[0070] Cloud platform 105 can receive model training requests and model parameter download requests, adjust model parameters for model training, distribute model topology structures and trained model parameters, and send road segment attribute information and real-time traffic information to mobile platforms 101, 102, and 103. For example, cloud platform 105 can serve as a backend management server, server cluster, or vehicle-to-everything (V2X) network.

[0071] It should be noted that the number of servers on mobile platforms, networks, and in the cloud is merely illustrative. Depending on implementation needs, any number of mobile platforms, networks, and cloud servers can be used.

[0072] Figure 2 The illustration shows a schematic diagram of an application scenario for storing video according to an embodiment of this application.

[0073] See Figure 2 The vehicle-mounted recording device, such as a dashcam, captures a continuous video stream, which includes at least two video segments captured within time period T1 and time period T2. Each video segment consists of multiple video frames.

[0074] If, based on the target object's speed and direction of movement relative to the mobile platform, the probability of a collision between the target object and the vehicle is low or virtually nonexistent, then the accident prediction result is "no accident is possible." In this case, only the video data captured within time period T1 can be stored in a cache. For example, the cache can be volatile memory, which has a higher read / write speed than non-volatile memory, but will experience more frequent data overwriting.

[0075] If, within time period T2, the target object's speed and direction of movement relative to the mobile platform indicate a high probability of collision with the vehicle, or if a collision has already occurred, then the accident prediction result is "An accident is possible." In this case, the video data captured within time period T2 can be stored in non-volatile memory. This preserves useful video data and reduces the likelihood of it being overwritten.

[0076] In this embodiment, an accident prediction model is used to identify vehicle and pedestrian behaviors in the video stream, and based on this, the likelihood of an accident occurring is predicted in advance. If the probability of an accident is greater than a certain threshold, it can be determined that an accident is imminent, and recording permissions for non-volatile memory are granted. Otherwise, only the video stream is generated without storing it in the non-volatile memory, or the video stream is overwritten. This approach reduces the storage of useless video data while ensuring the preservation of useful video data.

[0077] Figure 3 A flowchart illustrating a method for storing video according to an embodiment of this application is shown schematically.

[0078] See Figure 3 This embodiment provides a method for storing video, which includes operations S310 to S330, as detailed below.

[0079] During operation of S310, video data is acquired, including environmental video data of the mobile platform.

[0080] In this embodiment, the mobile platform includes, but is not limited to, any one of the following: a vehicle, a robot, a ship, or an aircraft. For example, video data can be obtained by a camera device installed on the vehicle. Specifically, the camera device can be a dashcam, etc.

[0081] The shooting device can be a monocular shooting device. Alternatively, a binocular shooting device can be used, and the two video data captured by the binocular shooting device can be fused together before accident prediction can be performed on the fused video data.

[0082] The environmental video data may include at least a portion of the images of the mobile platform itself, or it may not include images of the mobile platform itself. The environmental video data may include at least a portion of the images of the target object. The target object can be various man-made objects and non-man-made objects, such as buildings, vehicles, pedestrians, trees, etc.

[0083] In operating the S320, a trained accident prediction model is used to process video data and obtain accident prediction results.

[0084] Accident prediction models can be pre-trained models capable of determining whether a target object in the input video data is likely to interfere with the mobile platform, or whether the target object has already interfered with the mobile platform. Accident prediction models include, but are not limited to: Linear Regression, Logistic Regression, Decision Trees, Naive Bayes, K-Nearest Neighbors, Support Vector Machines, Random Decision Forests (or Bagging), and Neural Networks.

[0085] In some embodiments, the accident prediction model may employ deep neural networks. For example, the accident prediction model may extract features from each video frame in the input video data to obtain a feature map. Then, based on the feature map, or the feature map combined with other features, it may predict whether the mobile platform will interfere with the target object. As another example, the accident prediction model may extract features from each video frame in the input video data to obtain a feature map. Then, it may determine candidate boxes corresponding to the target object from the feature map. Next, based on the candidate boxes, it may obtain the feature data of the target object from the feature map. Finally, based on the feature data of the target object, or the feature data of the target object combined with other features, it may predict whether the mobile platform will interfere with the target object.

[0086] In operation S330, if the accident prediction result indicates that an accident has occurred on the mobile platform, then video data for a preset time period prior to the current moment is written to a non-volatile memory. For example, the non-volatile memory includes, but is not limited to, at least one of the following: erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc.

[0087] Training data can be categorized into training data for various scenarios. An accident prediction model can be a combination of multiple accident prediction sub-models trained separately on training data for different scenarios. Alternatively, an accident prediction model can be obtained by training the same accident prediction model using training data for different scenarios. Scenarios can include: highway grade, accident-prone areas, residential areas, scenic spots, etc. Different scenarios can correspond to different probability thresholds; for example, the probability threshold for accident-prone areas can be lower, while the probability threshold for highways can be higher.

[0088] It should be noted that the technical solution of this application is well applicable to scenarios where a vehicle is moving and the video is captured by a camera. In such scenarios, the vehicle may be moving at high speed, and the target object in the captured video frame moves rapidly. It is necessary to be able to quickly predict whether a traffic accident will occur from the video frame. In this scenario, mobile terminal devices with limited computing power need to quickly identify the target object from the video and determine whether the target object will interfere with the mobile platform.

[0089] The following is an example of an accident prediction model.

[0090] Figure 4 The diagram illustrates a structural schematic of an accident prediction model according to an embodiment of this application.

[0091] See Figure 4 The accident prediction model includes: a feature acquisition module, a candidate box acquisition module, a candidate feature acquisition module, and a recognition module.

[0092] The feature acquisition module is used to obtain feature maps from video frames in the video data.

[0093] The candidate bounding box acquisition module generates K×M candidate bounding boxes based on M pixels of the feature map, and selects N candidate bounding boxes that include the target object from the K×M candidate bounding boxes, where K is a preset number of candidate bounding boxes for a single pixel, and N and M are integers greater than 1. For example, the value of N is less than the value of K×M.

[0094] For example, for a pixel in a feature map, its corresponding K candidate boxes are a pre-defined set of candidate boxes. These candidate boxes can encompass all possible targets in the image, but their positions may not be entirely accurate, and it's unknown which candidate box contains the target object. Therefore, fine-tuning and classification of the candidate boxes are necessary.

[0095] The candidate feature acquisition module is used to obtain candidate features corresponding to each of the N candidate bounding boxes based on the feature map. Specifically, since there is a correspondence between pixels in the video frame and pixels in the feature map, a candidate bounding box for a specific region in the feature map can be mapped to a certain region in the video frame, and the corresponding candidate features can be determined. The number of candidate features is much smaller than the feature map, and it can significantly reduce the amount of noise data in the video frame (image data unrelated to traffic accidents, such as ground image data, blank image data, etc.), which helps improve the recognition effect.

[0096] The recognition module is used to determine the accident prediction result of the candidate box corresponding to the candidate features based on the candidate features. For example, the recognition module may include a fully connected layer to reduce the dimensionality of the feature data for a certain candidate box and obtain the accident prediction result for that candidate box.

[0097] Figure 5 The illustration shows a schematic diagram of a feature extraction module according to an embodiment of this application.

[0098] See Figure 5 The feature acquisition module includes: multiple convolutional pairs, and pooling layers set between some adjacent convolutional pairs.

[0099] For example, each convolutional pair includes an adjacent convolutional layer 1 and an activation layer 2. The convolutional kernel size of convolutional layer 1 is 3, the feature map padding width is 1, and the stride is 1. The pooling layer has a convolutional kernel size of 2, a feature map padding width of 0, and a stride of 2.

[0100] Figure 5 The diagram shows four pooling layers (3). The feature extraction module can include more or fewer pooling layers (3). There can be one or more convolutional pairs between two pooling layers (3). One or more convolutional pairs can precede the first pooling layer (3). The feature extraction module uses these multiple convolutional pairs and pooling layers (3) to extract features from video frames and output feature maps.

[0101] In one specific embodiment, video frames can be scaled to a fixed size X*Y, and then the scaled image is input into the feature acquisition module. The feature acquisition module may contain 6 convolutional layers, 6 activation function layers (such as ReLU function), and 4 pooling layers. The parameters of convolutional layer 1 and pooling layers are shown below.

[0102] Convolutional layer parameters: kernelsize=3, padding=1, stride=1.

[0103] Pooling layer parameters: kernelsize=2, padding=0, stride=2.

[0104] Here, `padding=1` makes the resolution of the video frame (X+2)*(Y+2). After convolution with a 3x3 kernel, the resolution of the output matrix is ​​X*Y. These convolutional layer parameter settings ensure that the input image and output matrix of the convolutional layer have the same size.

[0105] Furthermore, the parameters of the pooling layers cause each X*Y matrix after passing through a pooling layer to become (X / 2)*(Y / 2). The length and width of the output matrix of the pooling layer are both reduced to half the size of the input. Therefore, a video frame with a resolution of X*Y, after passing through four pooling layers, will have an output matrix of (X / 16)*(Y / 16). Thus, there is a correspondence between pixels in the feature map and pixels in the video frame. Here, 16 is 2 to the power of 4.

[0106] Figure 6 A schematic diagram illustrating a series of candidate boxes according to an embodiment of this application is shown.

[0107] See Figure 6 This method can generate multiple candidate boxes for a specific pixel in a feature map based on a pre-defined series of candidate boxes. For example, the candidate box sizes can be scales = [8, 16, 32], and the scaling ratios can be [0.5 / 1, 1 / 1, 2 / 1]. Each pixel in the feature map corresponds to 9 possible candidate box combinations. Assuming the feature map shape is [c, W, V], there are W*V pixels. Accordingly, all candidate boxes can be represented as [W, V, 9*4]. Each candidate box is represented as [x1, y1, x2, y2], where each value corresponds to a pixel in the video frame. This allows for the rapid generation of combined candidate boxes corresponding to each pixel in the feature map, resulting in a total of 9 candidate boxes. It should be noted that the number of candidate box types and the scaling ratios can be determined based on expert experience or usage results, and are not limited here. For example, the scaling ratios can also be [1 / 3, 1 / 2, 1 / 1, 2 / 1, 3 / 1], etc.

[0108] It should be noted that for a feature map with a resolution of X*Y, X*Y*9 candidate boxes can be generated, which involves a large amount of computation. To reduce the amount of computation, some candidate boxes can be selected from the X*Y*9 candidate boxes by means of specific classification or randomization.

[0109] In some embodiments, the candidate box acquisition module is specifically used to adjust at least one of the size parameters of K×M candidate boxes, and select N candidate boxes from the adjusted K×M candidate boxes as N detection boxes. Since the coverage of candidate boxes with preset sizes may be unsuitable, such as covering too large an area, leading to the introduction of noise, or the coverage of candidate boxes may be too small, causing the omission of some feature data for the target object, the coverage can be adjusted by adjusting the size of the candidate boxes.

[0110] Accordingly, the candidate feature acquisition module is specifically used to obtain the detection features corresponding to each of the N detection boxes based on the feature map.

[0111] The recognition module is specifically used to determine the accident prediction result based on the detection features and the corresponding detection box.

[0112] Figure 7 The illustration shows a schematic diagram of a candidate box obtaining module according to an embodiment of the present application.

[0113] See Figure 7 The candidate box acquisition module may include a first branch, a second branch, and a candidate box determination layer.

[0114] The first branch is used to classify the pixels of the feature map based on the activation function to obtain the category of the candidate box. For example, the categories include: including the target object and not including the target object. The number of pixels in the feature map is related to the number of pooling layers.

[0115] The second branch is used to calculate the offset relative to the candidate box.

[0116] The candidate box determination layer is used to determine candidate boxes, including the target object and the coverage of the candidate boxes on the feature map, based on the candidate box category and the candidate box offset.

[0117] Figure 7 The candidate bounding box acquisition module is divided into two lines. The upper line uses softmax to classify candidate bounding boxes by anchors, obtaining classifications that include and exclude the target object. The lower line calculates the offset of bounding box regression for the anchors to obtain accurate candidate bounding boxes. The final candidate bounding box determination layer combines candidate bounding boxes that include the target object with their corresponding offsets to obtain accurate candidate bounding boxes (detection boxes). Furthermore, it can discard candidate bounding boxes that are inappropriately sized (e.g., too small or too large). The candidate bounding box acquisition module implements localization functionality to determine which video frames (or feature maps) need to be identified.

[0118] Specifically, the second branch may include an adjustment module and an offset determination module.

[0119] The adjustment module is used to adjust at least one dimension of a candidate box (one of the K candidate boxes corresponding to each pixel in the feature map) according to multiple adjustment dimensions until the candidate box covers the target object and the coverage area of ​​the candidate box meets the area requirement. The multiple adjustment dimensions of the candidate box include at least one of top, bottom, left, and right. The area requirement can be that the candidate box covers the target object and has the smallest coverage area, or the candidate box...

[0120] The offset determination module is used to determine the offset of the candidate box corresponding to the pixel based on the average of the adjusted offsets of the K candidate boxes.

[0121] For example, a candidate bounding box is represented by [x1, y1, x2, y2], and the corresponding detection bounding box is represented by [X1, Y1, X2, Y2]. The adjustment module is used to determine the mapping relationship between [x1, y1, x2, y2] and [X1, Y1, X2, Y2]. This mapping relationship can be used to adjust each candidate bounding box to obtain the corresponding detection bounding box. The mapping relationship can be determined through model training or other methods.

[0122] Since the number of target objects in the feature map is limited and they are distributed in a specific area, the multiple detection boxes obtained by the adjustment module may correspond to one or more adjusted candidate boxes (detection boxes). The offset of the detection box for a target object can be determined by averaging.

[0123] This embodiment can quickly generate multiple candidate boxes for a single pixel in a feature map, and by adjusting the adjustment dimensions of the candidate boxes, a precise detection box can be easily obtained. This embodiment is applicable to target object detection and selection of the image region where the target object is located in video data captured during the movement of a mobile platform, making it easier to determine the probability of interference between the mobile platform and the target object.

[0124] The following is an illustrative description of the candidate feature acquisition module.

[0125] Figure 8 The illustration shows a schematic diagram of the candidate feature acquisition process according to an embodiment of this application.

[0126] See Figure 8 The candidate feature acquisition module overlays the feature map and the detection boxes from the candidate box determination layer at corresponding positions to obtain the detection features corresponding to each of the multiple detection boxes. This allows the recognition module to make predictions based only on the detection features, rather than on the entire feature map, effectively reducing computation and improving response speed.

[0127] In some embodiments, the identification module may include a fully connected layer, etc., for dimensionality reduction of the detection features to obtain the accident prediction result corresponding to the detection feature (or detection box).

[0128] Furthermore, considering that the probability of accidents occurring on mobile platforms varies under different environments—for example, the probability of accidents differs between accident-prone and accident-free sections of the same grade of highway—adding these dimensions to the input features could help improve the accuracy of accident prediction.

[0129] Specifically, the identification module is used to determine the accident prediction result of the candidate box corresponding to the candidate feature based on the candidate feature and the road segment attribute feature. The road segment attribute feature includes at least one of the following: highway grade, accident-prone road segment, school road segment or residential area road segment.

[0130] The road segment attribute features can be either the feature dimensions input to the feature acquisition module or the feature dimensions input to the recognition module; there is no limitation on this. Furthermore, road segment attribute features can be input by selecting a dataset, such as a training dataset for accident-prone road segment types, which can be used to specifically train an accident prediction model corresponding to those accident-prone road segment types.

[0131] Road segment attribute features can be features obtained externally, such as those from broadcasts, real-time traffic conditions, or third-party platforms. Road segment attribute features can also include road surface slippage (affected by weather). Highway grades can be divided into Class 1 to Class 5. Highway grades can also be categorized as rural roads, auxiliary roads, national highways, expressways, etc. For example, for the same video data, if the road segment attribute features include national highways, the accident prediction result indicates that no traffic accident will occur. However, for the same video data, if the road segment attribute features include scenic spots, the accident prediction result indicates that a traffic accident is highly likely to occur.

[0132] In some embodiments, the above method may further include the following operations.

[0133] For example, if the incident prediction result indicates that no incident has occurred on the mobile platform, a video stream for a preset time period prior to the current moment is generated, but this video stream is not stored. For instance, at least a portion of the video stream can be generated in a cache. Caches offer faster data read / write speeds but are more expensive. Therefore, the device's cache space is typically small, and cached video data is quickly overwritten by newly captured video data, making it unsuitable for long-term storage of useful video data. Furthermore, cached data is lost upon power-off. If the incident prediction result indicates that no incident has occurred on the mobile platform, then only the captured video data can be cached instead of being stored in non-volatile memory, improving data read / write speeds and saving non-volatile memory storage space.

[0134] For example, if the accident prediction result indicates that no accident has occurred on the mobile platform, then video data acquired after the current moment will overwrite video data acquired within a preset time period before the current moment. Specifically, a first-in, first-out (FIFO) approach can be used for video data storage to avoid overwriting the most recently stored video data.

[0135] In some embodiments, the probability of a mobile platform accident in the accident prediction results can be classified. For example, a collision has occurred (100% probability), a collision is extremely likely to occur (90%–99.9% probability), and a collision is relatively likely to occur (80%–89.9% probability). This facilitates the categorized storage of video data predicted to have resulted in a mobile platform accident, thereby further enhancing the security of the video data.

[0136] For example, collisions that have already occurred and collisions with a very high probability (90%–99.9%) correspond to the first storage area. Video data stored in the first storage area cannot be overwritten; it can only be cleared by user deletion operations. If the first storage area is full, video data that has already occurred and collisions with a very high probability can be stored in the second storage area.

[0137] Areas with a higher probability of collision (80%–89.9%) correspond to the second storage area. Video data stored in the second storage area can be overwritten, but is stored in a first-in, first-out (FIFO) manner.

[0138] In this embodiment, special video data storage rules can effectively improve the security and reliability of stored video data.

[0139] In some embodiments, the accident prediction model can be trained using the backpropagation algorithm. For details, please refer to neural network training methods.

[0140] In some embodiments, to improve the accuracy of the offsets of the obtained candidate boxes and to accurately obtain the feature data needed for accident prediction from the feature map, a two-stage training method can be adopted. Specifically, the base models for feature extraction and prediction can be trained first, and then the overall accident prediction model, consisting of the base model, the candidate box acquisition module, and the candidate feature acquisition module, can be trained. During the training of the base model, the required detection boxes can be externally input, such as by a trained model capable of obtaining detection boxes processing the training data to obtain the detection boxes.

[0141] Figure 9 A flowchart illustrating a training accident prediction model according to an embodiment of this application is shown.

[0142] See Figure 9The model training process may include operations S910 to S930.

[0143] In operation S910, a basic model consisting of a feature acquisition module, a candidate feature acquisition module, and a recognition module is trained based on the training data to obtain the trained basic model. The candidate boxes required by the candidate feature acquisition module can be obtained by processing the training data using a model that includes a trained candidate box acquisition module.

[0144] In the S920, the feature acquisition module, candidate box acquisition module, candidate feature acquisition module, and recognition module of the trained base model are sequentially spliced ​​together to obtain the accident prediction model to be trained.

[0145] When operating the S930, the training data is input into the feature acquisition module. The model parameters of the candidate box acquisition module are adjusted so that the output of the accident prediction model to be trained is close to the label data corresponding to the training data.

[0146] In addition, the above operations S910 to S930 can be repeated iteratively to improve the model prediction accuracy.

[0147] Specifically, the training data may include: video data collected by the mobile platform and label data corresponding to the video data, wherein the labels include at least one of the following: pedestrian lying down, pedestrian interfering with the mobile platform, interference between mobile platforms, pedestrian waving, pedestrian in running posture, and pedestrian in walking posture.

[0148] It should be noted that the accident prediction model can be trained offline or online, and can be trained in the cloud. Mobile platform video data storage devices (such as dashcams) can download the trained traffic accident prediction model's topology and parameters from the cloud to store video data on the mobile platform. Alternatively, the mobile platform can send video streams to the cloud, where the cloud can process the video data using the trained traffic accident prediction model to obtain accident prediction results. The cloud then sends (or broadcasts) these results to the mobile platform or video data storage device for video data storage.

[0149] Another aspect of this application provides a device for storing video.

[0150] Figure 10 A block diagram of a device for storing video according to an embodiment of this application is shown schematically.

[0151] See Figure 10The video storage device 1000 may include: a video data acquisition module 1010, a video data processing module 1020, and a video data storage module 1030.

[0152] The video data acquisition module 1010 is used to acquire video data, which includes environmental video data of the mobile platform.

[0153] The video data processing module 1020 is used to process video data using a trained accident prediction model, determine and output accident prediction results.

[0154] The video data storage module 1030 is used to write video data of a preset time period before the current moment into a non-volatile memory if the accident prediction result indicates that an accident has occurred on the mobile platform.

[0155] In some embodiments, the accident prediction model may include:

[0156] The feature acquisition module is used to extract feature maps from video frames in the video data;

[0157] The candidate box acquisition module is used to generate K×M candidate boxes based on M pixels of the feature map, and select N candidate boxes that include the target object from the K×M candidate boxes, where K is the preset number of candidate boxes for a pixel, and N and M are integers greater than 1.

[0158] The candidate feature acquisition module is used to obtain candidate features corresponding to each of the N candidate boxes based on the feature map.

[0159] The recognition module is used to determine the accident prediction result of the candidate box corresponding to the candidate feature based on the candidate feature.

[0160] In some embodiments, the candidate box obtaining module is specifically used to adjust at least one of the size parameters of the K×M candidate boxes, and select N candidate boxes from the adjusted K×M candidate boxes as N detection boxes.

[0161] The candidate feature acquisition module is specifically used to obtain the detection features corresponding to each of the N detection boxes based on the feature map.

[0162] The recognition module is specifically used to determine the accident prediction result based on the detection features and the corresponding detection box.

[0163] In some embodiments, the feature acquisition module includes:

[0164] Multiple convolutional pairs, and pooling layers disposed between partially adjacent convolutional pairs, wherein each convolutional pair includes an adjacent convolutional layer and an activation layer.

[0165] In some embodiments, the candidate box obtaining module includes:

[0166] The first branch is used to classify the pixels of the feature map based on the activation function to obtain the category of the candidate box. The category includes: including the target object and not including the target object. The number of pixels in the feature map is related to the number of pooling layers.

[0167] The second branch is used to calculate the offset relative to the candidate box.

[0168] The candidate box determination layer is used to determine candidate boxes, including the target object, and the coverage of the candidate boxes on the feature map based on the candidate box category and the candidate box offset.

[0169] In some embodiments, the second branch includes:

[0170] The adjustment module is used to adjust at least one of the multiple adjustment dimensions of any one of the K candidate boxes corresponding to each pixel in the feature map, until the candidate box covers the target object and the coverage area of ​​the candidate box meets the area requirement. The multiple adjustment dimensions of the candidate box include at least one of the following: top, bottom, left and right.

[0171] The offset determination module is used to determine the offset of the candidate box corresponding to the pixel based on the average of the adjusted offsets of the K candidate boxes.

[0172] In some embodiments, the identification module is specifically used to determine the accident prediction result of the candidate box corresponding to the candidate feature based on the candidate feature and the road segment attribute feature. The road segment attribute feature includes at least one of the following: highway grade, scenic road segment, accident-prone road segment, school road segment or residential area road segment.

[0173] Regarding the apparatus 1000 in the above embodiments, the specific methods by which each module and unit performs operations have been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0174] Another aspect of this application provides an electronic device.

[0175] Figure 11 A block diagram of an electronic device according to an embodiment of this application is shown schematically.

[0176] See Figure 11 The electronic device 1100 includes a memory 1110 and a processor 1120.

[0177] The processor 1120 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0178] Memory 1110 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 1120 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 1110 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1110 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0179] The memory 1110 stores executable code, which, when processed by the processor 1120, can cause the processor 1120 to execute part or all of the methods described above.

[0180] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0181] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0182] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of storing a video, the method comprising: The method comprises: obtaining video data, wherein the video data comprises environment video data of a mobile platform; processing the video data by using a trained accident prediction model to obtain an accident prediction result; if the accident prediction result indicates that the mobile platform has an accident, writing video data of a preset time period before a current time into a non-volatile memory; wherein the accident prediction model comprises: a feature obtaining module configured to obtain a feature map from a video frame in the video data; a candidate box obtaining module configured to generate K*M candidate boxes based on M pixels of the feature map, and select N candidate boxes including a target object from the K*M candidate boxes, wherein K is a preset number of candidate boxes for one pixel, and N and M are integers greater than 1; a candidate feature obtaining module configured to obtain candidate features corresponding to the N candidate boxes respectively based on the feature map; wherein the candidate feature obtaining module comprises a plurality of convolution pairs and a pooling layer arranged between part of adjacent convolution pairs, each convolution pair comprises an adjacent convolution layer and an activation layer; the candidate feature obtaining module comprises a first branch configured to classify pixels of the feature map based on an activation function to obtain a category of the candidate box, the category of the candidate box comprises a target object and a non-target object, and the number of pixels of the feature map is related to the number of the pooling layers; a second branch configured to calculate an offset of the candidate box; and a candidate box determination layer configured to determine the candidate box including the target object and a coverage range of the candidate box for the feature map based on the category of the candidate box and the offset of the candidate box; an identification module configured to determine an accident prediction result of the candidate box corresponding to the candidate feature based on the candidate feature.

2. The method of claim 1, wherein, The candidate box obtaining module is specifically configured to adjust at least one of size parameters of the K*M candidate boxes, and select N candidate boxes from the adjusted K*M candidate boxes as N detection boxes; The candidate feature obtaining module is specifically configured to obtain detection features corresponding to the N detection boxes respectively based on the feature map; The identification module is specifically configured to determine an accident prediction result of the detection box corresponding to the detection feature based on the detection feature.

3. The method of claim 1, wherein, The second branch comprises: an adjustment module configured to, for any one of K candidate boxes corresponding to each pixel in the feature map, adjust at least one of a plurality of adjustment dimensions of the candidate box to obtain an offset of the candidate box, wherein the adjusted candidate box covers the target object and the coverage area meets an area requirement, and the plurality of adjustment dimensions of the candidate box comprise at least one of up, down, left and right; an offset determination module configured to determine the offset of the candidate box corresponding to the pixel based on a mean value of the offsets of the adjusted K candidate boxes.

4. The method of claim 1, wherein, The identification module is specifically configured to determine an accident prediction result of the candidate box corresponding to the candidate feature based on the candidate feature and a road segment attribute feature, and the road segment attribute feature comprises at least one of a highway grade, a scenic road segment, a high-accident road segment, a school road segment or a residential road segment.

5. The method according to any one of claims 2 to 4, characterized in that, The accident prediction model is trained in the following manner: training a basic model composed of the feature obtaining module, the candidate feature obtaining module and the identification module based on the training data, to obtain a trained basic model, wherein the candidate frame required by the candidate feature obtaining module is obtained by processing the training data by a model including a trained candidate frame obtaining module; sequentially concatenating the feature obtaining module in the trained basic model, the candidate frame obtaining module, the candidate feature obtaining module in the trained basic model and the identification module in the trained basic model, to obtain an accident prediction model to be trained; inputting the training data into the feature obtaining module, and adjusting the model parameters of the candidate frame obtaining module, so that the output of the accident prediction model to be trained approaches the label data corresponding to the training data.

6. The method of claim 5, wherein, The training data includes video data collected by a mobile platform and label data corresponding to the video data, wherein the label includes at least one of the following: a pedestrian lying down, a pedestrian interfering with a mobile platform, interference between mobile platforms, a pedestrian waving, a pedestrian in a running posture, and a pedestrian in a walking posture.

7. The method according to any one of claims 1 to 4, characterized in that, Further comprising: if the accident prediction result indicates that the mobile platform has not had an accident, generating a video stream within a preset time period before the current time, and not storing the video stream; or if the accident prediction result indicates that the mobile platform has not had an accident, covering the video data within a preset time period before the current time with video data obtained after the current time.

8. An apparatus for storing a video, the apparatus comprising: including: a video data obtaining module configured to obtain video data, wherein the video data includes environmental video data of a mobile platform; a video data processing module configured to process the video data using a trained accident prediction model, determine and output an accident prediction result; a video data storage module configured to, if the accident prediction result indicates that the mobile platform has had an accident, write video data of a preset time period before the current time into a non-volatile memory; wherein the accident prediction model includes: a feature obtaining module configured to obtain a feature map from a video frame in the video data; a candidate frame obtaining module configured to generate K×M candidate frames based on M pixels of the feature map, and select N candidate frames including a target object from the K×M candidate frames, wherein K is a preset number of candidate frames for one pixel, and N and M are integers greater than 1; The candidate feature obtaining module is configured to obtain candidate features corresponding to the N candidate boxes respectively based on the feature map; wherein, the candidate feature obtaining module comprises a plurality of convolution pairs and a pooling layer arranged between adjacent convolution pairs, each convolution pair comprises a convolution layer and an activation layer arranged adjacently; the candidate feature obtaining module comprises a first branch configured to classify pixels of the feature map based on an activation function to obtain a category of the candidate box, the category of the candidate box comprises a target object and a non-target object, and a number of pixels of the feature map is related to a number of the pooling layers; a second branch configured to calculate an offset of the candidate box; and a candidate box determination layer configured to determine a candidate box comprising the target object and a coverage range of the candidate box for the feature map based on the category of the candidate box and the offset of the candidate box. The identification module is configured to determine an accident prediction result of the candidate box corresponding to the candidate feature based on the candidate feature.

9. An electronic device, comprising: The method comprises: a processor; and a memory having stored thereon executable code that, when executed by the processor, causes the processor to perform the method according to any one of claims 1-7. ​

Images