Airborne media data auditing method and system based on deep learning model, and medium
By deploying multiple deep learning framework models using container technology on the server and using pre-trained container models for media data identification, the deployment problem of limited hardware resources in traditional technology is solved, and efficient and accurate airborne media audit is achieved.
Patent Information
- Application Number
- CN202411634373.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-03-07
AI Technical Summary
Traditional technology for media auditing based on deep learning networks focuses on many models, resulting in more deep learning frameworks. With limited hardware resources, it is impossible to simultaneously compatible with the deployment of multiple deep learning framework models in one server.
By using container technology, multiple deep learning framework models are deployed in the same server, and through pre-trained container models, the multimedia video frames and image data are identified and matched, thereby achieving fast and accurate content classification, recognition and labeling.
It improves the accuracy of airborne media review in civil aviation, saves hardware costs, simplifies project deployment, improves convenience, and reduces the workload and error rate of manual review.
Smart Images
Figure CN119580154A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data recognition technology, and in particular to an airborne media data review method, system and medium based on a deep learning model. Background Art
[0002] According to relevant statistics, there are currently an average of 14,000 flights per day in China, carrying an average of 2 million passengers. Some of these flights provide in-flight entertainment media services. How to ensure more efficient and secure review and release of in-flight multimedia video image content, and to ensure that there are no risk behaviors such as violations of laws and regulations, commercial infringements, and negative public opinion, has become an issue that airlines are increasingly concerned about and urgently need to solve.
[0003] At present, most of the airlines' media review process is done manually. A large amount of video and image content requires a lot of manpower to review and verify it one by one. The quality and status of the reviewers also affect the quality of media review and release, and it is impossible to ensure that the media review and release can be carried out safely and effectively. Summary of the invention
[0004] The technical problem to be solved by the present invention is: the traditional technology of media review based on deep learning network, but because of the large number of focus models, resulting in a large number of deep learning frameworks, under the condition of limited hardware resources, it is impossible to simultaneously support the deployment of multiple deep learning framework models in one server; the purpose of the present invention is to provide an airborne media data review method, system and medium based on the deep learning model, and to improve the method and system architecture on the basis of the existing media review technology. This scheme applies the deep learning model deployment and media identification mechanism to the field of civil aviation airborne media, improves the accuracy of civil aviation airborne media review, and at the same time, through the use of container technology, can flexibly deploy multiple deep learning framework models in the same server according to the hardware resource conditions, saving hardware costs and improving the convenience of engineering deployment.
[0005] The present invention is achieved through the following technical solutions:
[0006] This solution provides an airborne media data review method based on a deep learning model, including:
[0007] Collecting airborne media data and preprocessing the airborne media data;
[0008] The pre-processed airborne media data is input into the container model for identification; the container model includes a plurality of trained container models of interest; the pre-processed airborne media data is input into each container model of interest in turn for comparison and identification; during the identification process, the container models are each run in a corresponding deep learning framework container environment;
[0009] Organize and output the recognition results.
[0010] Working principle of this solution: Traditional technology for media review based on deep learning networks, but because there are more models to focus on, there are more deep learning frameworks. In the case of limited hardware resources, it is impossible to simultaneously support the deployment of multiple deep learning framework models in one server; the purpose of the present invention is to provide an airborne media data review method, system and medium based on a deep learning model, and to improve the method and system architecture on the basis of the existing media review technology. This solution applies deep learning model deployment and media identification mechanism to the field of civil aviation airborne media to improve the accuracy of civil aviation airborne media review. At the same time, through the use of container technology, multiple deep learning framework models can be flexibly deployed in the same server according to hardware resources, saving hardware costs and improving the convenience of engineering deployment.
[0011] This solution uses a pre-trained container model to identify and match multimedia video frames and image data. If material within the monitoring range appears, the media information, location, hit data, etc. can be actively marked, so as to quickly and accurately classify, identify and mark the content of a large amount of airborne media. After the identification is completed, a complete identification analysis report is formed, reducing the workload and error rate of manual review; subsequent reviewers only need to review these suspected problem materials, greatly reducing the workload of manual review.
[0012] A further optimization scheme is that the collecting of airborne media data and preprocessing of the airborne media data include the following method:
[0013] S1, collects onboard media data;
[0014] S2, determining whether the current onboard media data is picture data or video data, if it is picture data, directly proceeding to step S3; if it is video data, performing video frame extraction processing on the video data to obtain picture data, and then proceeding to step S3;
[0015] S3, first normalizes the size and resolution of the image data, and then performs enhancement and denoising.
[0016] A further optimization scheme is that the pre-processed airborne media data is input into the container model for identification; including the method:
[0017] Obtain pre-processed airborne media data, and take each airborne media data as a recognition task;
[0018] Get the number of GPU resources n, randomly grab m recognition tasks, m≤n; input the captured recognition tasks into different attention container models for parallel recognition, and each recognition process occupies one GPU resource;
[0019] After the current recognition task has traversed all the attention container models, the recognition result of the current recognition task is output.
[0020] Further optimization solutions include focusing on the identification process of the container model including:
[0021] Obtain the image data of the current recognition task and calculate the matching value between each image data and the focus container model;
[0022] A matching degree threshold is set. When the matching degree value is higher than the matching degree threshold, the current image data is determined to be a hit target, and the hit position is marked in the current image data.
[0023] A further optimization scheme is that after the current recognition task has traversed all the focus container models, the recognition result of the current recognition task is output; including the method:
[0024] Get the recognition results of all image data of the current recognition task;
[0025] If the image data of the current recognition task hits the target, the current recognition task is written into the media recognition result database; if the image data of the current recognition task does not hit the target, the current recognition task is discarded.
[0026] A further optimization solution is that the deep learning model framework includes: PyTorch framework, TensorFlow framework, Caffe framework and MxNet framework.
[0027] The present solution also provides an airborne media data review system based on a deep learning model, which is used to implement the above-mentioned airborne media data review method based on a deep learning model; the system comprises:
[0028] A collection module, used to collect airborne media data;
[0029] Messaging middleware to enable communication with the onboard media data review system for deep learning models;
[0030] The preprocessing module is used to preprocess the airborne media data;
[0031] A recognition module, used for inputting the pre-processed airborne media data into the container model for recognition;
[0032] The output module is used to organize and output the recognition results.
[0033] A further optimization scheme is that the media identification module includes: an identification task management layer, a container model layer, an image resource layer, and a GPU resource layer;
[0034] The recognition task management layer is used to construct the recognition task and the allocation mechanism of GPU resources and image resources during the recognition process; the recognition task management layer is also used to perform data integration analysis on the recognition results to generate an analysis report;
[0035] The container model layer includes a plurality of trained focus container models;
[0036] The image resource layer includes multiple deep learning model frameworks;
[0037] The GPU resource layer includes multiple GPU resources.
[0038] A further optimization plan includes a media review interaction interface for media review human-computer interaction.
[0039] The present solution also provides a computer-readable medium having a computer program stored thereon, and the computer program is executed by a processor to implement the above-mentioned onboard media data review method based on the deep learning model.
[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0041] 1. The present invention provides an airborne media data review method, system and medium based on a deep learning model. Based on the existing media review technology, improvements are made to the method and system architecture. This solution applies deep learning model deployment and media identification mechanism to the field of civil aviation airborne media to improve the accuracy of civil aviation airborne media review. At the same time, through the use of container technology, multiple deep learning framework models can be flexibly deployed in the same server according to hardware resource conditions, saving hardware costs and improving the convenience of engineering deployment.
[0042] 2. The present invention provides an airborne media data review method, system and medium based on a deep learning model; this solution uses a pre-trained container model to identify and match multimedia video frames and image data. If material within the monitoring range appears, the media information, location, hit data, etc. can be actively marked. Subsequent reviewers only need to review these suspected problem materials, greatly reducing the workload of manual review and fatigue error rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:
[0044] Figure 1It is a flowchart of the airborne media data review method based on the deep learning model;
[0045] Figure 2 Schematic diagram of the pretreatment process;
[0046] Figure 3 It is a schematic diagram of the analysis process of the recognition results;
[0047] Figure 4 This is a schematic diagram of the structure of the airborne media data review system based on the deep learning model;
[0048] Figure 5 This is a schematic diagram of the structure of the media identification module;
[0049] Figure 6 This is a schematic diagram of the working principle of the media identification module;
[0050] Figure 7 This is a schematic diagram of the media review interaction interface. DETAILED DESCRIPTION
[0051] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.
[0052] At present, most of the media review process of airlines is done manually. A large amount of video image content requires a large number of people to check and verify one by one. The quality and status of the reviewers also affect the quality of media review and release, and it is impossible to ensure that the media review and release can be carried out safely and effectively. Even if there is a technology for media review based on deep learning networks, due to the large number of models, there are many deep learning frameworks. In the case of limited hardware resources, it is impossible to simultaneously support the deployment of multiple deep learning framework models in one server. In view of this, this solution provides the following embodiments to solve the above technical problems:
[0053] Example 1
[0054] This embodiment provides an onboard media data review method based on a deep learning model, such as Figure 1 As shown, including:
[0055] Step 1: collect airborne media data and pre-process the airborne media data; Figure 2 As shown, this step specifically includes the following method:
[0056] S1, collects onboard media data;
[0057] S2, determining whether the current onboard media data is picture data or video data, if it is picture data, directly proceeding to step S3; if it is video data, performing video frame extraction processing on the video data to obtain picture data, and then proceeding to step S3;
[0058] S3, first normalizes the size and resolution of the image data, and then performs enhancement and denoising.
[0059] Convert image data and video frame extraction image data parameters into standard data suitable for input container model.
[0060] Step 2: input the preprocessed airborne media data into a container model for identification; the container model includes a plurality of trained container models of interest; the preprocessed airborne media data is sequentially input into each container model of interest for comparison and identification; during the identification process, each container model runs in a corresponding deep learning framework container environment;
[0061] This step specifically includes the following methods:
[0062] Obtain the preprocessed airborne media data, and take each airborne media data as a recognition task; (each recognition task is performed based on an independent media as a basic unit, where each airborne media data includes a picture or a video, and the video contains multiple extracted key frame pictures).
[0063] Get the number of GPU resources n, randomly grab m recognition tasks, m≤n; input the captured recognition tasks into different attention container models for parallel recognition, and each recognition process occupies one GPU resource;
[0064] After the current recognition task has traversed all the attention container models, the recognition result of the current recognition task is output.
[0065] The identification process of the container model in the above steps includes:
[0066] Obtain the image data of the current recognition task and calculate the matching value between each image data and the focus container model;
[0067] A matching degree threshold is set. When the matching degree value is higher than the matching degree threshold, the current image data is determined to be a hit target, and the hit position is marked in the current image data.
[0068] After the current recognition task has traversed all the focus container models, the recognition result of the current recognition task is output; Figure 3 As shown, including methods:
[0069] Get the recognition results of all image data of the current recognition task;
[0070] If the image data of the current recognition task hits the target, the current recognition task is written into the media recognition result database; if the image data of the current recognition task does not hit the target, the current recognition task is discarded.
[0071] The specific attention container model includes models that airlines may pay attention to: blood, pornography, tattoos, self-immolation, fighting, screen distortion, etc. The attention container model is pre-trained;
[0072] Step 3: Organize and output the recognition results.
[0073] The collecting of airborne media data and preprocessing of the airborne media data,
[0074] The deep learning model framework includes: PyTorch framework, TensorFlow framework, Caffe framework and MxNet framework. The recognition results are integrated and analyzed to form a complete analysis report, including which video, which frames, and which monitoring targets were hit, for interface query and manual verification.
[0075] Example 2
[0076] This embodiment provides an airborne media data review system based on a deep learning model, which is used to implement the airborne media data review method based on a deep learning model described in Example 1; Figure 4 As shown, the system comprises:
[0077] A collection module, used to collect airborne media data;
[0078] Messaging middleware to enable communication with the onboard media data review system for deep learning models;
[0079] The preprocessing module is used to preprocess the airborne media data;
[0080] The message middleware obtains the preprocessing tasks triggered externally, parses the messages, and the preprocessing module queues multiple media that need to be preprocessed. If the media is image data, it directly performs image preprocessing, standardizes the image size and resolution, and then adds image enhancement, image denoising and other technologies to improve the image information. If the media is video data, it needs to first go through video frame extraction, usually extracting video key frames, and then process the video frame images accordingly.
[0081] The recognition module is used to input the pre-processed airborne media data into the container model for recognition; Figure 5 and Figure 6As shown, the media identification module includes: an identification task management layer, a container model layer, an image resource layer and a GPU resource layer; the identification task management layer is used to construct the identification task and the allocation mechanism of GPU resources and image resources during the identification process; the identification task management layer is also used to perform data integration analysis on the identification results to generate an analysis report;
[0082] The container model layer includes a plurality of trained focus container models;
[0083] The image resource layer includes multiple deep learning model frameworks;
[0084] The GPU resource layer includes multiple GPU resources.
[0085] The output module is used to organize and output the recognition results.
[0086] The deep learning model frameworks include: PyTorch framework, TensorFlow framework, Caffe framework and MxNet framework; each deep learning model framework depends on different environments. In order to expand the support of the media review system for the deep learning model framework, this solution can support the rapid deployment of pre-trained models of each framework based on Docker container technology; the media recognition task input by the preprocessing module is obtained from the middleware, and the pictures to be recognized in each video are fed into each model in turn for recognition. During the process, the execution status of each model task and the GPU usage will be comprehensively considered, and the recognition task scheduling mechanism will be dynamically adjusted to ensure that more tasks can be carried out in parallel and all models and GPU resources are maximized.
[0087] In the media recognition module, each recognition task is performed based on an independent media unit, such as a picture or a video (which contains many extracted key frames). After the recognition task is pushed to all the concerned container models in turn, the recognition data will be written back to the media recognition result information. After all the recognitions are completed, the recognition results can be screened to find the pictures, videos, locations and other information that match the target, and write the result data into the database for data acquisition and processing in the subsequent media result data management process to form a complete recognition report.
[0088] like Figure 7 As shown, it also includes a media review interaction interface for human-computer interaction in media review. It provides human-computer interaction functions, obtains and selects media to be reviewed from the media pool management module, can view the progress and results of media review, and provides review summary report query and other functions. This solution is based on the message subscription distribution mechanism, decouples the dependency of functional modules to the greatest extent, reduces the message communication volume, and provides business communication functions for each functional module. It manages the added or deleted media and provides media query access and editing functions for the interface.
[0089] Example 3
[0090] This embodiment provides a computer-readable medium on which a computer program is stored. The computer program is executed by a processor to implement the airborne media data review method based on a deep learning model as described in Embodiment 1. Specifically, the following steps are performed:
[0091] Step 1: collecting airborne media data and preprocessing the airborne media data;
[0092] Step 2: input the preprocessed airborne media data into a container model for identification; the container model includes a plurality of trained container models of interest; the preprocessed airborne media data is sequentially input into each container model of interest for comparison and identification; during the identification process, the container model is deployed in a plurality of deep learning model frameworks;
[0093] Step 3: Organize and output the recognition results.
[0094] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. An airborne media data review method based on a deep learning model, characterized in that: include: Collecting airborne media data and preprocessing the airborne media data; The pre-processed airborne media data is input into the container model for recognition; The container model includes a plurality of trained container models of interest; the pre-processed airborne media data is sequentially input into each container model of interest for comparison and identification; during the identification process, each container model runs in a corresponding deep learning framework container environment; Organize and output the recognition results.
2. The airborne media data review method based on deep learning model according to claim 1, characterized in that: The collecting of airborne media data and preprocessing of the airborne media data include: S1, collects onboard media data; S2, determine whether the current onboard media data is image data or video data, if it is image data, directly proceed to step S3; If it is video data, the video data is subjected to video frame extraction processing to obtain image data, and then the process proceeds to step S3; S3, first normalizes the size and resolution of the image data, and then performs enhancement and denoising.
3. The airborne media data audit method based on deep learning model according to claim 2, characterized in that: The pre-processed airborne media data is input into the container model for identification; Included methods: Obtain pre-processed airborne media data, and take each airborne media data as a recognition task; Get the number of GPU resources n, randomly grab m recognition tasks, m≤n; input the captured recognition tasks into different attention container models for parallel recognition, and each recognition process occupies one GPU resource; After the current recognition task has traversed all the attention container models, the recognition result of the current recognition task is output.
4. The airborne media data review method based on deep learning model according to claim 3 is characterized in that: The identification process of the focus container model includes: Obtain the image data of the current recognition task and calculate the matching value between each image data and the focus container model; A matching degree threshold is set. When the matching degree value is higher than the matching degree threshold, the current image data is determined to be a hit target, and the hit position is marked in the current image data.
5. The airborne media data review method based on deep learning model according to claim 4, characterized in that: After the current recognition task has traversed all the focus container models, the recognition result of the current recognition task is output; Included methods: Get the recognition results of all image data of the current recognition task; If the image data of the current recognition task hits the target, the current recognition task is written into the media recognition result database; if the image data of the current recognition task does not hit the target, the current recognition task is discarded.
6. The airborne media data audit method based on deep learning model according to claim 1, characterized in that: The deep learning model framework includes: PyTorch framework, TensorFlow framework, Caffe framework and MxNet framework.
7. The airborne media data review system based on the deep learning model is characterized by: Used to implement the airborne media data review method based on the deep learning model described in claim 5; the system includes: A collection module, used to collect airborne media data; Messaging middleware to enable communication with the onboard media data review system for deep learning models; The preprocessing module is used to preprocess the airborne media data; A recognition module, used for inputting the pre-processed airborne media data into the container model for recognition; The output module is used to organize and output the recognition results.
8. The airborne media data review system based on deep learning model according to claim 7, characterized in that: The media identification module includes: an identification task management layer, a container model layer, an image resource layer, and a GPU resource layer; The recognition task management layer is used to construct the recognition task and the allocation mechanism of GPU resources and image resources during the recognition process; the recognition task management layer is also used to perform data integration analysis on the recognition results to generate an analysis report; The container model layer includes a plurality of trained focus container models; The image resource layer includes multiple deep learning model frameworks; The GPU resource layer includes multiple GPU resources.
9. The airborne media data review system based on deep learning model according to claim 6, characterized in that: It also includes a media review interaction interface for media review human-computer interaction.
10. A computer readable medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the onboard media data review method based on a deep learning model as described in any one of claims 1-5.