Video data processing method, device, electronic device and storage medium

Abstract

An embodiment of the present application provides a video data processing method, device, electronic device, and computer storage medium, wherein the method includes: obtaining a video stream, performing frame processing on the video stream, and obtaining an image to be detected in each frame; performing target detection on each frame of the image to be detected to obtain a detection result; the detection result includes at least one position frame and the confidence of the at least one position frame; performing target tracking on at least one position frame in each frame of the image to be detected to obtain at least one tracking trajectory; selecting a preset number of target position frames based on the confidence of each position frame in the at least one tracking trajectory; and using a trained target classifier to perform target classification on the image corresponding to each target position frame to obtain a classification result of the video stream.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video processing, and relate to but are not limited to a video data processing method, device, electronic device, and computer storage medium. Background Art

[0002] Multi-Object Tracking (MOT) systems in video streams are widely used in the field of computer vision. Multi-object tracking refers to the task of tracking all detected targets in a video. Currently, most video stream MOT systems are based on a track-by-detect design framework. However, under this design framework, due to the widespread imbalance of detection datasets, the detector's ability to recognize confusing targets is weak, resulting in many missed detections and low target recognition accuracy. Summary of the Invention

[0003] Based on the problems in the related art, the embodiments of the present application provide a video data processing method and device.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a method for processing video data, including:

[0006] Acquire a video stream, perform frame processing on the video stream, and obtain an image to be detected in each frame;

[0007] Performing target detection on each frame of the image to be detected to obtain a detection result; the detection result includes at least one position frame and a confidence level of the at least one position frame;

[0008] Performing target tracking on at least one position frame in each frame of the image to be detected to obtain at least one tracking track; selecting a preset number of target position frames according to the confidence level of each position frame in the at least one tracking track;

[0009] The trained target classifier is used to perform target classification on the image corresponding to each target position frame to obtain a classification result of the video stream.

[0010] An embodiment of the present application provides a video data processing device, including:

[0011] An acquisition module is used to acquire a video stream, perform frame processing on the video stream, and obtain an image to be detected in each frame;

[0012] A detection module, configured to perform target detection on each frame of the image to be detected to obtain a detection result; the detection result includes at least one position frame and a confidence level of the at least one position frame;

[0013] a tracking module configured to perform target tracking on at least one position frame in each frame of the image to be detected to obtain at least one tracking track; and select a preset number of target position frames based on the confidence level of each position frame in the at least one tracking track;

[0014] The classification module is used to use the trained target classifier to perform target classification on the image corresponding to each target position box to obtain the classification result of the video stream.

[0015] An embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned video data processing method is implemented.

[0016] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned video data processing method is implemented.

[0017] The video data processing method, device, electronic device and computer storage medium provided in the embodiment of the present application obtain a video stream, perform frame processing on the video stream, obtain each frame of the image to be detected; perform target detection on each frame of the image to be detected to obtain a detection result; the detection result includes at least one position frame and the confidence of the at least one position frame; perform target tracking on at least one position frame in each frame of the image to be detected to obtain at least one tracking trajectory; select a preset number of target position frames based on the confidence of each position frame in the at least one tracking trajectory; use the trained target classifier to perform target classification on the image corresponding to each target position frame to obtain the classification result of the video stream. It can be seen that in the embodiment of the present application, on the basis of target detection on each frame of the image to be detected in the video stream, target tracking is performed on each detected position frame, and the images corresponding to the position frames in the tracking trajectory whose confidence meets the conditions are further refined and classified, so that the classification results can be made more refined and the confusing targets can be better identified; furthermore, the problem of missed detection caused by the imbalance of the detection data set in the related art is solved, and the accuracy of target classification is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a video data processing method provided in an embodiment of the present application;

[0019] Figure 2 This is a flow chart of training a target classifier according to an embodiment of the present application;

[0020] Figure 31 is a flow chart of determining the classification result of a video stream provided by an embodiment of the present application;

[0021] Figure 4 This is a flow chart of using the deep learning framework TensorRT for reasoning acceleration provided in an embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of another process of video data processing provided by an embodiment of the present application;

[0023] Figure 6 Schematic diagram of the structure of the video data processing device provided in an embodiment of the present application;

[0024] Figure 7 It is a schematic diagram of the composition structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0026] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it will be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which the embodiments of this application pertain. The terms used in the embodiments of this application are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0027] In related technologies, most video stream MOT systems are based on a track-by-detect design framework. However, under this design framework, due to the widespread imbalance of detection datasets, the detector's ability to recognize confusing targets is weak, resulting in many missed detections, resulting in low target recognition accuracy and affecting the performance of the video stream MOT system.

[0028] In order to solve the above problems, the following embodiments are proposed.

[0029] In some embodiments of the present application, the video data processing method can be implemented using a processor in a video data processing device, and the above-mentioned processor can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0030] Figure 1 A flow chart of a video data processing method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the process may include:

[0031] Step S101: Acquire a video stream, perform frame processing on the video stream, and obtain an image to be detected in each frame.

[0032] Exemplarily, a video stream refers to the transmission of video data, which includes multiple video frames that are continuous in time sequence. Here, there is no limitation on the type of video stream. For example, the video stream can be a video stream captured in real time by a camera, or it can be a pre-cached offline video stream.

[0033] In an embodiment of the present application, after obtaining the video stream, the video stream needs to be framed; here, the frame processing means dividing the video stream into individual image frames to obtain an image to be detected in each frame; illustratively, the frame processing can be achieved by decoding the video stream.

[0034] It can be understood that if the acquired video stream includes 50 temporally continuous video frames, 50 frames of images to be detected can be obtained after frame processing.

[0035] Step S102: performing target detection on each frame of the image to be detected to obtain a detection result; the detection result includes at least one position frame and a confidence level of the at least one position frame.

[0036] Exemplarily, each frame of the image to be detected may include at least one target object to be tracked and identified, where the at least one may be one or more. In the embodiment of the present application, there is no limitation on the type of the target object, which may be determined according to the actual scenario. For example, in a vehicle tracking scenario, the target object may be a certain model of vehicle; in a pedestrian tracking scenario, the target object may be a pedestrian with certain characteristics.

[0037] In an embodiment of the present application, in order to more effectively detect and identify the categories of each target object, after obtaining each frame of the image to be detected, target detection is first performed on each frame of the image to be detected to obtain a position frame of at least one target object and the corresponding confidence level of each position frame.

[0038] Exemplarily, a trained target detector can be used to perform target detection on each frame of the image to be detected; here, there is no limitation on the type of target detector, for example, it can be a convolutional neural network (CNN) model, etc.

[0039] For example, each location box has a corresponding confidence level, which can take a value between 0 and 1; wherein the location box can be a rectangular box with a certain height and width, which is used to indicate the size and position of the target object. The location box is also called a detection box or a bounding box.

[0040] It can be understood that if each frame of the image to be detected includes one target object, a position frame of the target object and the confidence of the position frame will be detected; if each frame of the image to be detected includes multiple target objects, the position frames of multiple target objects and the confidence of multiple position frames will be detected.

[0041] Step S103 : performing target tracking on at least one position frame in each frame of the image to be detected to obtain at least one tracking track; and selecting a preset number of target position frames according to the confidence level of each position frame in the at least one tracking track.

[0042] Exemplarily, target tracking is performed on at least one position frame in each frame of the image to be detected to obtain at least one tracking trajectory, which may include: when each frame of the image to be detected includes one target object, target tracking can be performed on the position frame of the detected target object to obtain one tracking trajectory; when each frame of the image to be detected includes multiple target objects, multi-target tracking can be performed on the position frames of the detected multiple target objects to obtain multiple tracking trajectories at the same time.

[0043] Accordingly, after obtaining at least one (including the above-mentioned one or more) tracking track, a preset number of target position frames may be selected according to the confidence level of each position frame in the at least one tracking track.

[0044] In some embodiments, selecting a preset number of target position frames according to the confidence of each position frame in the at least one tracking trajectory may include: selecting the target position frame with the highest confidence from the position frames included in each tracking trajectory in the at least one tracking trajectory.

[0045] For example, for the position frames included in each tracking track in at least one tracking track, the confidence levels of these position frames can be sorted in descending order, and the position frame with the highest confidence level is selected from the sorting results as the target position frame; that is, a target position frame can be obtained for each tracking track.

[0046] It can be understood that the value of the preset number corresponds to the number of tracking tracks; that is, when at least one tracking track includes three tracking tracks, a target position frame can be obtained for each of the three tracking tracks. In this case, the value of the preset number is 3.

[0047] Step S104: Use the trained target classifier to perform target classification on the image corresponding to each target position frame to obtain a classification result of the video stream.

[0048] In the embodiment of the present application, after obtaining the target position frame according to the above steps, the trained target classifier is used to classify the image corresponding to the target position frame. In some embodiments, the trained target classifier can be obtained by pre-training the target classifier. Figure 2 This is a flow chart of training a target classifier according to an embodiment of the present application. Figure 2 As shown, the target classifier can be trained by the following steps:

[0049] Step S201: obtaining an initial classification data set, performing an incremental operation on the initial classification data set to obtain a classification data set;

[0050] Step S202: Use the classification data set to train the target classifier to obtain the trained target classifier.

[0051] Exemplarily, the initial classification dataset may be some existing datasets used to achieve target classification. The embodiment of the present application does not limit the type of the initial classification dataset. For example, it may be an ImageNet dataset or other types of datasets.

[0052] Exemplarily, performing the incremental operation on the initial classification data set may include: performing the incremental operation on the initial classification data set according to a set condition; the set condition includes at least one of an illumination condition and a geometric transformation condition.

[0053] Here, the lighting conditions may refer to the lighting change conditions caused by different light sources; wherein the lighting changes will not only affect the color of the image, but also affect the texture detail information of the image; the geometric transformation may include spatial position transformations such as rotation, translation, and scaling; it can be understood that by performing incremental operations on the initial classification data set through at least one of the lighting conditions and geometric transformation conditions, the data volume of the initial classification data set can be greatly enriched.

[0054] For example, since the initial classification data set is incrementally operated, the image data in the incremental classification data set is more evenly distributed. Therefore, the target classifier trained based on the incremental classification data set can ensure the accuracy of subsequent target classification results.

[0055] In the embodiment of the present application, after the target classifier is trained using the classification data set after the incremental operation, the trained target classifier can be obtained. The target classifier can be used to perform target classification on the image corresponding to the target position box to obtain the classification result of the video stream. Figure 3 FIG. 1 is a flow chart of determining the classification result of a video stream provided by an embodiment of the present application, such as Figure 3 As shown, in some embodiments, step S104 can be implemented by the following steps:

[0056] Step S301 : performing a clipping operation on each target position frame according to position information of each target position frame on the corresponding image to be detected, to obtain each target area image.

[0057] In some embodiments, the position information may be the four position coordinates of the target position frame, and the height and width of the target position frame may be determined based on the four position coordinates; then, based on the height and width of the target position frame, the target position frame on the image to be detected is cropped, and thus, an image corresponding to the target position frame, i.e., a target area image, is obtained.

[0058] Step S302: Use the trained target classifier to perform target classification on each target area image to obtain a classification result of the video stream.

[0059] For example, after each target region image is obtained through the interception operation, a trained target classifier is used to perform target classification on each target region image to obtain a classification result for the video stream. Here, the classification result may include a target category and a corresponding confidence level for the target category. That is, for each target category included in the classification result, there is a corresponding confidence level. Thus, if the classification result includes only one target category, the classification result of the video stream may include one target category and a corresponding confidence level for the target category. If the classification result includes multiple target categories, the classification result of the video stream may include multiple target categories and corresponding confidence levels for each of the multiple target categories.

[0060] It should be noted that, compared with the above-mentioned target detector for target detection, the target classifier is a small network for target classification. The embodiment of the present application performs target detection on each frame of the image to be detected, and uses the detected position frame of the target object to track the target, and uses a small target classifier to perform fine classification of the image corresponding to the area of ​​the position frame with the best confidence. It can greatly improve the target classification accuracy while having a small impact on the system processing delay.

[0061] Exemplarily, in order to reduce system processing delay and improve model processing speed, in some embodiments, the above method may also include: before using the trained target classifier to perform target classification on the image corresponding to the target position box (i.e., the target area image), using the deep learning framework TensorRT to perform inference acceleration on the trained target classifier; optionally, before performing the above-mentioned target detection, the deep learning framework TensorRT may also be used to perform inference acceleration on the corresponding target detector.

[0062] For example, Figure 4 This is a flow chart of using the deep learning framework TensorRT for reasoning acceleration provided by the embodiment of the present application, such as Figure 4 As shown in the figure, for the target detector trained according to the detection dataset, the target platform classifier can be obtained by performing inference acceleration through the deep learning framework TensorRT; similarly, after using the incremental classification dataset to train the model and obtain the trained target classifier, the target platform classifier can be obtained by performing inference acceleration through the deep learning framework TensorRT.

[0063] Here, the principle of using the deep learning framework TensorRT for inference acceleration is: by analyzing the structured information of the original model (corresponding to the above-mentioned target classifier and target detector), the neural network graph is optimized and the data type is converted on the target platform. The weights and biases of the original model remain unchanged, so the detection accuracy and recognition accuracy of the model after inference acceleration will not change. It can be seen that using the deep learning framework TensorRT for inference acceleration can improve the processing speed while ensuring that the detection accuracy and recognition accuracy remain unchanged.

[0064] The video data processing method, device, electronic device and computer storage medium provided in the embodiment of the present application obtain a video stream, perform frame processing on the video stream, obtain each frame of the image to be detected; perform target detection on each frame of the image to be detected to obtain a detection result; the detection result includes at least one position frame and the confidence of the at least one position frame; perform target tracking on at least one position frame in each frame of the image to be detected to obtain at least one tracking trajectory; select a preset number of target position frames based on the confidence of each position frame in the at least one tracking trajectory; use the trained target classifier to perform target classification on the image corresponding to each target position frame to obtain the classification result of the video stream. It can be seen that in the embodiment of the present application, on the basis of target detection on each frame of the image to be detected in the video stream, target tracking is performed on each detected position frame, and the image corresponding to each position frame with the highest confidence in each tracking trajectory is further refined and classified, so that the classification result can be made more refined and the confusing target can be better identified; furthermore, the problem of missed detection caused by imbalanced detection data sets in the related art is solved, and the accuracy of target classification is improved.

[0065] In order to better reflect the purpose of this application, further explanation is given based on the above embodiments of this application.

[0066] Figure 5 This is another flow chart of video data processing provided by an embodiment of the present application. Figure 5As shown, after inputting a real-time video stream, the trained target detector is first used to perform target detection on each frame of the image to be detected in the real-time video stream to obtain the location frame of each target object and the confidence of each location frame; then, the target is tracked on each location frame in each frame of the image to be detected to obtain the tracking trajectory corresponding to each target object; then, from the location frames included in each tracking trajectory, the target location frame with the highest confidence is selected, and the image corresponding to each target location frame is screenshotted to obtain each target area image, and then the target classifier is used to classify each target area image, and finally the classification result of each target object is obtained; wherein the classification result includes the target category and the confidence corresponding to the target category. It should be noted that for the input real-time video stream, in addition to outputting the classification result of each target object, the tracking trajectory of each target object is also output.

[0067] It can be seen that in the embodiment of the present application, the recognition accuracy of the confused target is improved by first performing target detection and then classifying the detection results; this is because the classification data set is easier to increment than the detection data set to make its distribution more balanced, and then, the target classifier obtained by training the incremental classification data set is used to further classify the detection results, which can make the classification results more refined, better identify confused targets, and ensure the accuracy of the subsequent MOT system.

[0068] Figure 6 FIG. 1 is a schematic diagram of the structure of a video data processing device provided in an embodiment of the present application. Figure 6 As shown, the video data processing device 400 includes: an acquisition module 401, a detection module 402, a tracking module 403 and a classification module 404, wherein:

[0069] The acquisition module 401 is used to acquire a video stream, perform frame processing on the video stream, and obtain an image to be detected in each frame;

[0070] A detection module 402 is configured to perform target detection on each frame of the image to be detected to obtain a detection result; the detection result includes at least one position frame and a confidence level of the at least one position frame;

[0071] The tracking module 403 is configured to perform target tracking on at least one position frame in each frame of the image to be detected to obtain at least one tracking track; and select a preset number of target position frames based on the confidence level of each position frame in the at least one tracking track;

[0072] The classification module 404 is configured to use the trained target classifier to perform target classification on the image corresponding to each target position frame to obtain a classification result of the video stream.

[0073] In some embodiments, the tracking module 403 is configured to select a preset number of target position frames according to the confidence level of each position frame in the at least one tracking trajectory, including:

[0074] A target position frame with the highest confidence is selected from the position frames included in each tracking track in the at least one tracking track.

[0075] In some embodiments, the apparatus further comprises a training module, wherein the training module is configured to:

[0076] Obtaining an initial classification data set, and performing an incremental operation on the initial classification data set to obtain a classification data set;

[0077] The target classifier is trained using the classification data set to obtain the trained target classifier.

[0078] In some embodiments, the training module is configured to perform incremental operations on the initial classification data set, including:

[0079] An incremental operation is performed on the initial classification data set according to a set condition; the set condition includes at least one of an illumination condition and a geometric transformation condition.

[0080] In some embodiments, the classification module 404 is configured to use a trained target classifier to perform target classification on the image corresponding to each target position box to obtain a classification result of the video stream, including:

[0081] performing a clipping operation on each target position frame according to position information of each target position frame on the corresponding image to be detected, to obtain each target area image;

[0082] The trained target classifier is used to perform target classification on each target area image to obtain a classification result of the video stream.

[0083] In some embodiments, the classification result includes: a target category and a confidence level corresponding to the target category.

[0084] In some embodiments, the video data processing apparatus further includes an acceleration module, wherein the acceleration module is configured to:

[0085] Before using the trained target classifier to perform target classification on the image corresponding to the target position box, the deep learning framework TensorRT is used to accelerate the inference of the trained target classifier.

[0086] In actual applications, the above-mentioned acquisition module 401, detection module 402, tracking module 403, classification module 404, training module and acceleration module can all be implemented by a processor located in an electronic device, and the processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0087] In addition, the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional modules.

[0088] If the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the part that contributes to the relevant technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0089] Specifically, the computer program instructions corresponding to a video data processing method in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the computer program instructions corresponding to a video data processing method in the storage medium are read or executed by an electronic device, any one of the video data processing methods in the aforementioned embodiments is implemented.

[0090] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0091] Correspondingly, an embodiment of the present application provides an electronic device, Figure 7 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 7As shown, the electronic device 500 includes at least: a processor 501 and a computer-readable storage medium 502 configured to store executable instructions, wherein the processor 501 generally controls the overall operation of the electronic device. The computer-readable storage medium 502 is configured to store instructions and applications executable by the processor 501, and can also cache data to be processed or processed by the processor 501 and various modules in the electronic device 500, which can be implemented using flash memory (FLASH) or RAM.

[0092] The embodiment of the present application provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application, for example, Figure 1 The method shown.

[0093] In some embodiments, the storage medium can be a computer-readable storage medium, such as a ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disk read-only memory (CD-ROM); it can also be various devices including one or any combination of the above memories.

[0094] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0095] By way of example, executable instructions may, but need not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). By way of example, executable instructions may be deployed for execution on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0096] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

[0097] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0098] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed.

[0099] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A video data processing method, the method comprising: Acquire a video stream, perform frame processing on the video stream, and obtain an image to be detected in each frame; Performing target detection on each frame of the image to be detected to obtain a detection result; The detection result includes at least one position frame and a confidence level of the at least one position frame; Performing target tracking on at least one position frame in each frame of the image to be detected to obtain at least one tracking track; selecting a preset number of target position frames according to the confidence level of each position frame in the at least one tracking track; Using a trained target classifier to perform target classification on the image corresponding to each target position box to obtain a classification result of the video stream; wherein the trained target classifier is obtained by training the target classifier based on the incremental classification data set; The step of selecting a preset number of target position frames according to the confidence level of each position frame in the at least one tracking trajectory includes: The confidence of each position frame included in each tracking track in the at least one tracking track is sorted in descending order, and the target position frame with the highest confidence is selected from each position frame included in each tracking track according to the sorting result.

2. The method according to claim 1, wherein the target classifier is trained by the following steps: Obtaining an initial classification data set, and performing an incremental operation on the initial classification data set to obtain a classification data set; The target classifier is trained using the classification data set to obtain the trained target classifier.

3. The method according to claim 2, wherein the incremental operation on the initial classification data set comprises: Performing incremental operations on the initial classification data set according to set conditions; The setting condition includes at least one of an illumination condition and a geometric transformation condition.

4. The method according to claim 1, wherein the performing target classification on the image corresponding to each target position box using a trained target classifier to obtain a classification result of the video stream comprises: performing a clipping operation on each target position frame according to position information of each target position frame on the corresponding image to be detected, to obtain each target area image; The trained target classifier is used to perform target classification on each target area image to obtain a classification result of the video stream.

5. The method according to any one of claims 1 to 4, wherein the classification result comprises: The target category and the confidence level corresponding to the target category.

6. The method according to any one of claims 1 to 4, further comprising: Before using the trained target classifier to perform target classification on the image corresponding to the target position box, the deep learning framework TensorRT is used to accelerate the inference of the trained target classifier.

7. A video data processing device, comprising: An acquisition module is used to acquire a video stream, perform frame processing on the video stream, and obtain an image to be detected in each frame; A detection module, configured to perform target detection on each frame of the image to be detected to obtain a detection result; the detection result includes at least one position frame and a confidence level of the at least one position frame; a tracking module configured to perform target tracking on at least one position frame in each frame of the image to be detected to obtain at least one tracking trajectory; and select a preset number of target position frames based on the confidence of each position frame in the at least one tracking trajectory; wherein the target position frame is the position frame with the highest confidence among each position frame included in each tracking trajectory of the at least one tracking trajectory, and is determined based on a ranking result of the confidences; the ranking result is obtained by ranking the confidences of each position frame in descending order; A classification module is used to use a trained target classifier to perform target classification on the image corresponding to each target position box to obtain a classification result of the video stream; wherein, the trained target classifier is obtained by training the target classifier based on the incremental classification data set.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the program.

9. A computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images