Video structured processing method, system and device and storage medium

Abstract

The invention discloses a video structuring processing method, system and device and a storage medium, the method is based on a video structuring system, the video structuring system comprises a video decoding module, an image processing module and a model processing module, the video decoding module comprises a plurality of decoding sub-modules, the image processing module comprises a plurality of image processing sub-modules, and the model processing module comprises a plurality of image processing sub-modules. The model processing module comprises a plurality of model processing sub-modules; the method comprises the following steps: S1, acquiring a video data processing task, wherein the video data processing task carries a data processing flow chart and to-be-processed task data; s2, based on the data processing flow chart, calling a corresponding decoding sub-module, a graphic processing sub-module and a model processing sub-module in sequence to process the to-be-processed task data; and S3, integrating processing results of the called decoding sub-module, the graph processing sub-module and the model processing sub-module to obtain a task processing result. According to the scheme, computing resources can be saved, and the resource utilization rate is increased.

Description

Technical Field

[0001] The present application relates to the technical field of video data processing, and in particular to a video structured processing method, system, device and storage medium. Background Art

[0002] The video structuring method mainly processes video data intelligently and extracts key information to assist upper-level business applications. Its main functional modules include video data decoding, image processing, model reasoning, etc. However, upper-level business applications often process a large number of video data channels, and due to different business requirements, the model functions they rely on are also different. In addition, even for a single channel of video data, due to the complexity of business requirements, multiple different models are required to support it.

[0003] In existing video structuring systems, the processing of video data and the scheduling of related models are often preset in advance. The video frame data is processed in sequence through these preset fixed processing flows, and finally a structured result is output. In actual scenarios, there will be additional calls to some functional modules that do not need to be called, resulting in a waste of computing power and limiting the upper limit of the system's processing capacity. This also means that it is impossible to adapt to the diversified needs of upper-level businesses, and it is impossible to dynamically adjust the processing flow for newly accessed video data, which brings challenges to the actual implementation process and application. Summary of the invention

[0004] In order to solve the above problems, the present application provides a video structuring processing method, system, device and storage medium.

[0005] In a first aspect, an embodiment of the present application provides a video structuring processing method, the method is based on a video structuring system, the video structuring system includes a video decoding module, an image processing module, and a model processing module, the video decoding module includes a plurality of decoding submodules, the image processing module includes a plurality of image processing submodules, and the model processing module includes a plurality of model processing submodules;

[0006] The method comprises:

[0007] S1: Acquire a video data processing task, wherein the video data processing task carries a data processing flow chart and task data to be processed;

[0008] S2: Based on the data processing flow chart, the corresponding decoding submodule, graphics processing submodule, and model processing submodule are called in sequence to process the task data to be processed;

[0009] S3: Integrate the processing results of the called decoding submodule, graphics processing submodule, and model processing submodule to obtain the task processing result.

[0010] In some embodiments, the data processing flow chart is a directed acyclic graph, including a number of process nodes;

[0011] After obtaining the video processing task, the method further includes:

[0012] Verify whether all process nodes in the data processing flow chart match the decoding submodule or the graphics processing submodule or the model processing submodule;

[0013] Verifying whether the data processing flow chart complies with directed acyclicity;

[0014] If the verification results are all passed, proceed to the subsequent steps.

[0015] In some embodiments, based on the data processing flow chart, calling the corresponding decoding submodule, graphics processing submodule, and model processing submodule in sequence to process the task data to be processed includes:

[0016] S21: Identify a process node with an in-degree of 0 in the data processing flow chart as a target node;

[0017] S22: Identify a target submodule corresponding to the target node, where the target submodule is one of all decoding submodules, graphic processing submodules, and model processing submodules;

[0018] S23: deleting the target node from the data processing flow chart to obtain an updated data processing flow chart;

[0019] S24: sending the updated data processing flow chart and the current task data to the target submodule, obtaining the processing result of the target submodule by processing the current task data, and updating the task data based on the processing result of the target submodule;

[0020] S25: Acquire the updated task data;

[0021] S21 to S25 are repeatedly executed until there is no process node with an in-degree of 0 in the updated data processing flow chart.

[0022] In some embodiments, the decoding submodule includes an H264 decoding submodule, a HEVC decoding submodule, and an MJPEG decoding submodule.

[0023] In some embodiments, the image processing submodule includes a JPEG encoding submodule, an H264 encoding submodule, an HEVC encoding submodule, a color processing submodule, and a scaling and cropping submodule.

[0024] In some embodiments, the model processing submodule includes a human body detection submodule, a face recognition submodule, a vehicle detection submodule, a license plate recognition submodule, and a smoke detection submodule.

[0025] In a second aspect, an embodiment of the present application provides a video structuring system, including a task scheduling module, a message queue module, a video decoding module, an image processing module, a model processing module, and an information integration module, wherein the video decoding module includes a plurality of decoding submodules, the image processing module includes a plurality of image processing submodules, and the model processing module includes a plurality of model processing submodules, wherein:

[0026] The task scheduling module is used to obtain a video data processing task, wherein the video data processing task carries a data processing flow chart and task data to be processed;

[0027] The task scheduling module is also used to sequentially call the corresponding decoding submodule, graphics processing submodule, and model processing submodule to process the task data to be processed based on the data processing flow chart;

[0028] The information integration module is used to integrate the processing results of the called decoding submodule, graphics processing submodule, and model processing submodule to obtain the task processing result.

[0029] In a third aspect, an embodiment of the present application provides a vehicle-mounted control device, which is applied to new energy vehicles. The vehicle-mounted control device includes a processor and a memory, and the memory stores a computer program. When the processor executes the computer program, it implements the video structuring processing method described in the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a storage medium having a program stored thereon, and when the program is executed by a processor, it is used to implement the video structuring processing method as described in the first aspect.

[0031] The technical solution of this application has at least the following advantages:

[0032] 1. The video structured system calls the corresponding decoding submodule, graphics processing submodule, and model processing submodule in sequence to process the task data to be processed according to the data processing flowchart. For different data processing flowcharts, the decoding submodule, graphics processing submodule, and model processing submodule called by the video structured system will also be different, and no additional submodules will be called, which saves computing resources and improves resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 is a flowchart of a video structured processing method provided by an exemplary embodiment of the present application;

[0035] Figure 2 is a flowchart of step S2 in a video structuring processing method provided by an exemplary embodiment of the present application;

[0036] Figure 3 is a schematic diagram for reflecting a video decoding module and its corresponding message queue provided by an exemplary embodiment of the present application;

[0037] Figure 4 is a schematic diagram of a possible data processing flow chart provided by an exemplary embodiment of the present application;

[0038] Figure 5 This is an exemplary embodiment of the present application. Figure 3 A schematic diagram of the data processing flow chart after one update;

[0039] Figure 6 is a schematic diagram of another possible data processing flow chart provided by an exemplary embodiment of the present application;

[0040] Figure 7 is a system block diagram of a video structuring system provided by an exemplary embodiment of the present application;

[0041] Figure 8 It is a structural block diagram of a terminal device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0044] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0045] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0046] The present application provides a video structuring processing method, which is based on a video structuring system, which can be mounted in a server host, a computer terminal or a cloud server as the execution subject of the present method. Among them, the video structuring system includes several functional modules, which include a video decoding module, an image processing module and a model processing module. The video decoding module can decode the compressed data in the video stream into raw frame data for use by subsequent modules. In order to cope with different data types, the video decoding module includes several decoding submodules, such as H264 decoding submodule, HEVC decoding submodule, MJPEG decoding submodule, etc. The color space of the raw frame data obtained by decoding can be NV12, NV21 or BGR, etc. The image processing module is used to perform further image processing on the raw frame data, such as color conversion, scaling and cropping, encoding, etc. Correspondingly, the image processing module also includes several independent image processing submodules, such as JPEG encoding submodule, H264 encoding submodule, HEVC encoding submodule, color processing submodule, scaling and cropping submodule, etc. The model processing module is used to load the model file to perform image analysis on the original frame data after image processing, so as to realize the intelligent analysis function of various objects such as human body, human face, vehicle, etc. Correspondingly, the model processing module can include several model processing submodules, such as human body detection submodule, face recognition submodule, vehicle detection submodule, license plate recognition submodule, smoke detection submodule, etc. The video decoding module also includes a log module for recording log information for problem troubleshooting and backtracking.

[0047] Reference Figure 1 The video structuring processing method provided in the embodiment of the present application includes the following steps:

[0048] S1: Obtain a video data processing task, which carries a data processing flow chart and task data to be processed.

[0049] For example, the task data to be processed can be a video stream file obtained by a video acquisition device, and the data processing flowchart can be edited by a staff member through a customized computer or other equipment according to business needs and in combination with system functions. The data processing flowchart is used to define the processing flow of the task data to be processed. After receiving the video data processing task issued by the staff member through the upper-level business application, the video structured system will trigger subsequent processing.

[0050] Furthermore, the data processing flow chart in the above-mentioned can be a directed acyclic graph (DAG), which includes several process nodes, such as a start node, an intermediate node, and an end node. After obtaining the data processing flow chart, the video structuring system can first verify the data processing flow chart, and then perform subsequent processing after the verification is passed. Among them, a verification module can be configured in the video structuring system to perform a verification process, and the verification process can be as follows:

[0051] Verify whether all process nodes in the data processing flow chart match the decoding submodule or the graphics processing submodule or the model processing submodule.

[0052] Exemplarily, for any process node in the data processing flow chart, the video structuring system will verify whether it matches one of the decoding submodules, graphics processing submodules, or model processing submodules. For example, the verification can be completed by matching and confirming the name of the process node with each submodule name in turn. When the name of a process node is "ODInfer", the video structuring system will use the name to search for submodules whose names begin with "ODInfer" in all decoding submodules, graphics processing submodules, and model processing submodules. When the submodule whose name begins with "ODInfer" is identified, it means that the match is successful. When it is verified that all process nodes in the data processing flow chart match one of all decoding submodules, graphics processing submodules, and model processing submodules, it means that this verification has passed.

[0053] Verify whether the data processing flowchart complies with directed acyclicity.

[0054] Exemplarily, the video structuring system may verify whether the data processing flow graph complies with directed acyclicity.

[0055] If the verification results are all passed, proceed to the subsequent steps.

[0056] Exemplarily, when both of the above verifications are passed, it means that the data processing flow chart meets the processing requirements, and the video structuring system will execute subsequent processing steps.

[0057] S2: Based on the data processing flow chart, the corresponding decoding submodule, graphics processing submodule, and model processing submodule are called in sequence to process the task data to be processed.

[0058] The video structured system also includes a task scheduling module and a message queue module. The task scheduling module is used to schedule task data at different stages to corresponding submodules for processing according to the processing flow defined in the data processing flow chart. The message queue module includes several message queues. The video decoding module, the image processing module, and the model processing module each have a message queue corresponding to them. At the same time, all video decoding submodules share the message queue corresponding to the video decoding module. Figure 3 As shown, all image processing submodules share the message queue corresponding to the image processing module, and all model processing submodules share the message queue corresponding to the model processing module. Each message queue is used to cache the task data to be processed sent by the task scheduling module for the submodule in the corresponding functional module to call.

[0059] Exemplarily, the video structuring system calls the corresponding decoding submodule, graphics processing submodule, and model processing submodule in sequence to process the task data to be processed according to the data processing flowchart. For different data processing flowcharts, the decoding submodule, graphics processing submodule, and model processing submodule called by the video structuring system will also be different, and no additional submodules will be called, which saves computing resources and improves resource utilization. For example, if the data processing flowchart is as follows Figure 4 As shown, the decoding submodule called is the H264 decoding submodule, the graphics processing submodule called is the JPEG encoding submodule, and the model processing submodules called are the human body detection submodule and the smoke detection submodule respectively.

[0060] Further, see Figure 2 , this step S2 may specifically include the following processing flow:

[0061] S21: Identify a process node with an in-degree of 0 in the data processing flow chart as a target node.

[0062] Exemplarily, different types of process nodes have different in-degrees and out-degrees. Specifically, the in-degree of the initial node is 0 and the out-degree is 1, the in-degree and out-degree of the intermediate node are both 1, and the in-degree of the terminal node is 1 and the out-degree is 0. In order to process the process nodes in sequence, the video structuring system will first identify the process node with an in-degree of 0 in the data processing flow chart as the target node.

[0063] S22: Identify the target submodule corresponding to the target node, where the target submodule is one of all decoding submodules, graphic processing submodules, and model processing submodules.

[0064] Exemplarily, the video structuring system will identify the target submodule corresponding to the target node. The identification process can be as described in the matching process in S1, which will not be repeated here. Figure 4Taking the data processing flow chart in as an example, the first target node identified is "H264 decoding", and the corresponding target submodule is the H264 decoding submodule.

[0065] S23: Delete the target node from the data processing flow chart to obtain an updated data processing flow chart.

[0066] Exemplarily, the video structuring system deletes the target node from the data processing flow chart to obtain an updated data processing flow chart. Figure 4 As an example, after deleting the target node "H264 decoding", the updated data processing flowchart is as follows Figure 5 As shown. Figure 5 In the example, "JPEG Encoding" becomes the new initial node.

[0067] S24: Send the updated data processing flow chart and the current task data to the target submodule, obtain the processing result of the target submodule by processing the current task data, and update the task data based on the processing result of the target submodule.

[0068] Exemplarily, the video structuring system sends the updated data processing flow chart and the current task data to the message queue corresponding to the target submodule through the task scheduling module, so that the target submodule can call and process it. Figure 4 Taking the data processing flow chart in as an example, the current task data is the original video stream file. After the H264 decoding submodule as the target submodule decodes it, the original frame data obtained is the processing result. Afterwards, the video structuring system updates the original frame data to new task data for subsequent submodules to process.

[0069] S25: Obtain updated task data.

[0070] Exemplarily, the task scheduling module in the video structuring system obtains updated task data from the target submodule.

[0071] S21 to S25 are repeatedly executed until there is no process node with an in-degree of 0 in the updated data processing flow chart.

[0072] For example, Figure 4Taking the data processing flow chart in as an example, by repeatedly executing S21 to S25 in the above, the video structuring system sequentially calls the H264 decoding submodule to decode the original video stream file into the original frame data, calls the JPEG encoding submodule to encode the original frame data, calls the human body detection submodule to perform human body detection processing on the encoded original frame data, calls the smoke detection submodule to perform smoke detection processing on the original frame data that has been processed by human body detection, and obtains the processing results of each step. After deleting the final node "smoke detection", there is no process node with an in-degree of 0 in the data processing flow chart, and this step ends.

[0073] In this embodiment, Figure 4 The linear data processing flow chart in FIG. 1 is described. In other embodiments, the data processing flow chart may also be as follows: Figure 6 As shown, in Figure 6 In the process of "human body detection" and "smoke detection", the two process nodes are parallel. Figure 6 When processing the data processing flow chart in, after calling the JPEG encoding submodule to encode the original frame data, calling the human body detection submodule to perform human body detection processing on the encoded original frame data, and calling the smoke detection submodule to perform smoke detection processing on the encoded original frame data are two steps performed simultaneously, which helps to achieve balanced allocation of system resources.

[0074] S3: Integrate the processing results of the decoding submodule, the graphics processing submodule, and the model processing submodule to obtain the task processing result.

[0075] Exemplarily, the video structuring system integrates the processing results of the called decoding submodule, graphics processing submodule, and model processing submodule to obtain the task processing result. Afterwards, the video structuring system can upload the task processing result to the upper-layer business application.

[0076] The present application also provides a video structuring system, referring to Figure 7 , which includes a task scheduling module, a message queue module, a video decoding module, an image processing module, a model processing module, and an information integration module. The video decoding module includes several decoding submodules, the image processing module includes several image processing submodules, and the model processing module includes several model processing submodules, wherein:

[0077] The task scheduling module is used to obtain video data processing tasks, which carry data processing flow charts and task data to be processed;

[0078] The task scheduling module is also used to sequentially call the corresponding decoding submodule, graphics processing submodule, and model processing submodule to process the task data to be processed based on the data processing flow chart;

[0079] The information integration module is used to integrate the processing results of the called decoding submodule, graphics processing submodule, and model processing submodule to obtain the task processing results.

[0080] refer to Figure 8 The present application also provides a terminal device, which includes a processor and a memory.

[0081] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire device, and executes various functions of the device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but may be implemented separately through a chip.

[0082] Optionally, the processor implements the video structuring processing method provided by each of the above method embodiments when executing the program instructions in the memory.

[0083] The memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the device, etc.

[0084] Optionally, the present application also provides a storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the video structuring processing method provided by the above method embodiment.

[0085] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this application.

Claims

1. A video structuring processing method, characterized in that: The method is based on a video structuring system, which includes a video decoding module, an image processing module, and a model processing module. The video decoding module includes a plurality of decoding submodules, the image processing module includes a plurality of image processing submodules, and the model processing module includes a plurality of model processing submodules. The method comprises: S1: Obtain a video data processing task, wherein the video data processing task carries a data processing flow chart and task data to be processed; S2: Based on the data processing flow chart, the corresponding decoding submodule, graphics processing submodule, and model processing submodule are called in sequence to process the task data to be processed; S3: Integrate the processing results of the called decoding submodule, graphics processing submodule, and model processing submodule to obtain the task processing result.

2. The video structuring processing method according to claim 1, characterized in that: The data processing flow chart is a directed acyclic graph, including several process nodes; After obtaining the video processing task, the method further includes: Verify whether all process nodes in the data processing flow chart match the decoding submodule or the graphics processing submodule or the model processing submodule; Verifying whether the data processing flow chart complies with directed acyclicity; If the verification results are all passed, proceed to the subsequent steps.

3. The video structuring processing method according to claim 2, characterized in that: Based on the data processing flow chart, the corresponding decoding submodule, the graphics processing submodule, and the model processing submodule are sequentially called to process the task data to be processed, including: S21: Identify a process node with an in-degree of 0 in the data processing flow chart as a target node; S22: Identify a target submodule corresponding to the target node, where the target submodule is one of all decoding submodules, graphic processing submodules, and model processing submodules; S23: deleting the target node from the data processing flow chart to obtain an updated data processing flow chart; S24: sending the updated data processing flow chart and the current task data to the target submodule, obtaining the processing result of the target submodule by processing the current task data, and updating the task data based on the processing result of the target submodule; S25: Acquire the updated task data; S21 to S25 are repeatedly executed until there is no process node with an in-degree of 0 in the updated data processing flow chart.

4. The video structuring processing method according to claim 1, characterized in that: The decoding submodule includes an H264 decoding submodule, a HEVC decoding submodule, and an MJPEG decoding submodule.

5. The video structuring processing method according to claim 1, characterized in that: The image processing submodule includes a JPEG encoding submodule, an H264 encoding submodule, an HEVC encoding submodule, a color processing submodule, and a scaling and cropping submodule.

6. The video structuring processing method according to claim 1, characterized in that: The model processing submodule includes a human body detection submodule, a face recognition submodule, a vehicle detection submodule, a license plate recognition submodule, and a smoke detection submodule.

7. A video structuring system, characterized in that: It includes a task scheduling module, a message queue module, a video decoding module, an image processing module, a model processing module, and an information integration module. The video decoding module includes several decoding submodules, the image processing module includes several image processing submodules, and the model processing module includes several model processing submodules, wherein: The task scheduling module is used to obtain a video data processing task, wherein the video data processing task carries a data processing flow chart and task data to be processed; The task scheduling module is also used to sequentially call the corresponding decoding submodule, graphics processing submodule, and model processing submodule to process the task data to be processed based on the data processing flow chart; The information integration module is used to integrate the processing results of the called decoding submodule, graphics processing submodule, and model processing submodule to obtain the task processing result.

8. A terminal device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A storage medium having a program stored thereon, wherein when the program is executed by a processor, the program is used to implement the method according to any one of claims 1 to 6.

Images