Video structured system design engine, method, computer device, and medium

Abstract

In order to overcome the problem that the code and development logic cannot be reused when a video structured system is designed in the prior art, development time and manpower are greatly wasted, the application provides a video structured system design engine, a video structured system, a method, a computer device and a medium. The application discloses a video structured system design engine, which comprises an action processing component library, a modeling module and an execution module. The business process required in the video structured design is analyzed, and a plurality of action processing units with minimum action processing functions are designed. The action processing units are integrated in the action processing component library, and are selected and combined by a designer according to business requirements. The video structured system design engine disclosed by the application greatly simplifies the upper layer code of the video structured system, reduces the development time of the video structured application, reduces the development difficulty, improves the development efficiency and the debugging efficiency of the video structured application.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video structuring systems, in particular to the field of design engines of video structuring systems. BACKGROUND

[0002] With the progress of artificial intelligence technology, video structuring technology has been used in all aspects of modern life. For example, people are familiar with face recognition access control, face capture cameras, license plate capture cameras and other intelligent terminals, which all integrate video structuring systems to extract useful information from video files or real-time video streams, thus greatly facilitating people's work and life.

[0003] However, when developing and designing a video structuring system, the existing technology usually focuses on two main contents: one is data structure, and the other is business-specific process, including specific data extraction, result saving, optimization, heterogeneous computing, multi-thread synchronization, etc. It builds corresponding business modules based on specific businesses to realize video structuring systems with different platforms and functions. The data structure and business process in the video structuring system based on different targets and needs will be very different for different platforms (end devices, servers) and specific business types, resulting in almost no effective reuse of the code and development logic between them, which greatly wastes development time and manpower. SUMMARY

[0004] To overcome the problem that the code and development logic cannot be reused when designing a video structuring system in the prior art, which greatly wastes development time and manpower, the present application provides a video structuring system design engine, a video structuring system, a method, a computer device and a medium.

[0005] In one aspect, the present application discloses a video structuring system design engine, comprising the following modules:

[0006] An action processing component library, which includes a plurality of action processing units; the action processing units are used to perform corresponding action processing on the input data frame, and load the processing result in the data frame for output;

[0007] A modeling module, used to select action processing units from the action processing component library, and connect the selected action processing units as nodes to form a behavior graph according to business requirements;

[0008] An execution module, used to parse and execute the behavior graph, and generate a target video structuring system.

[0009] Further, the data frame comprises at least image information, currently detected object information, being tracked object information and being disappeared object information. The data frame is pre-configured for the flow of each node, and the result of the action processing unit of each node is left a position, so that the processed result can be output to the subsequent node through the data frame. The processing result of each node can be written into the data frame.

[0010] Further, the action processing unit comprises:

[0011] A video source reading unit is configured to read a video source to be structured, and output a data frame for the subsequent action processing unit;

[0012] A detector is configured to detect the image information in the input data frame, obtain the currently detected object information, and write the currently detected object information into the data frame and output;

[0013] A mapper is configured to perform mapping operation on the currently detected object information or the being tracked object information in the input data frame, generate a mapping result, and write the mapping result into the data frame and output;

[0014] A filter is configured to perform filtering on the input data frame by using a filtering function with the object information as the independent variable and a logic value as the dependent variable, write the filtering result into the data frame and output;

[0015] A matcher is configured to receive a plurality of object sets from the input data frame, and perform logical operation on the object sets, and output a matching set after the logical operation;

[0016] A matching bridge unit is configured to connect the matcher with other nodes.

[0017] Through the above basic action processing unit, most of the video structured application design can be completed, and the application development and design requirements of the development designer can be met. Further, the development time and difficulty of the video structured application are reduced, and the development efficiency and debugging efficiency of the video structured application are improved.

[0018] Further, the matching bridge unit comprises a matching input bridge unit and a matching output bridge unit.

[0019] The matching input bridge unit is configured to connect the matcher with the action processing unit of the previous input node, and the matching output bridge unit is configured to connect the matcher with the action processing unit of the subsequent output node.

[0020] Further, the action processing unit further comprises a selector, which is configured to select object information meeting the index requirement and corresponding data frame output in a time period from a plurality of data frames input in the time period.

[0021] Further, the selector comprises the following:

[0022] a full cycle selector, which is configured to select data frame output corresponding to object information meeting the optimal index requirement in the index requirement from a time period in which the target object appears to disappear;

[0023] a selection cycle selector, which is configured to select data frame output meeting the optimal index in the index requirement from a time period in which the target object appears to disappear;

[0024] a minimum threshold selector, which is configured to select all frame outputs meeting the index requirement better than the minimum threshold in the index requirement from a time period in which the target object appears to disappear;

[0025] a bubble selector, which is configured to select data frame output meeting the index requirement better than the optimal index selected last time in the index requirement from a time period in which the target object appears to disappear.

[0026] Further, the action processing unit further comprises a frame divider, which is configured to divide data frames to processing branches formed by different nodes.

[0027] Further, the action processing unit further comprises a saving unit, which is configured to save data frames output in the previous node.

[0028] Further, the action processing unit further comprises a tracking unit, which is configured to track the object to be tracked in the input data frame to output the tracking result.

[0029] Further, the action processing component library further comprises a secondary development component with a specific business function, which is generated by combining the action processing unit.

[0030] The second aspect of the present application provides a video structured system design method, which applies the video structured system design engine to design a video structured system; the video structured system design method comprises the following steps:

[0031] receiving a process setting instruction, and setting a business process of a target video structured system according to the process setting instruction;

[0032] receiving an action selection instruction, and selecting an action processing unit associated with the business process according to the action selection instruction;

[0033] connecting the action processing units associated with the business process as nodes into a behavior graph that implements the business process;

[0034] parsing and executing the behavior graph to generate a target video structured system.

[0035] The third aspect of the present application provides a computer device, comprising a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, wherein the computer readable instructions are the readable instructions generated by the video structured system design engine and the readable instructions generated by the video structured system design method.

[0036] The fourth aspect of the present application provides one or more readable storage media, wherein the readable storage media stores the codes involved in the video structured system design engine and the video structured system design method.

[0037] The video structured system design engine, the computer device and the storage media disclosed in the present application analyze the business process needed in the video structured design, design a plurality of action processing units with minimum action processing function, integrate the action processing units in an action processing component library, and select and combine the action processing units according to the business requirements for use by the designers; the modeling module connects the action processing units as nodes into a behavior graph that can implement the business function; the execution module can correctly parse and execute the behavior graph to finally generate a target video structured system. The video structured system design engine decouples all actions involved in the video structured design into independent action processing units, connects the action processing units to form a behavior graph, so that one behavior graph can represent a complete video structured process, and the target video structured system meeting the design requirements can be obtained after execution. In this way, the tedious data structure management, heterogeneous computing, multi-thread synchronization and other transactions can be processed by the underlying execution engine, which enables the developers to focus on the description of specific business without worrying about numerous underlying transactions, greatly simplifies the upper code of the video structured, reduces the development time and difficulty of the video structured application, and improves the development and debugging efficiency of the video structured application. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a schematic diagram of the video structured system design engine principle in Example 1 provided by the specific embodiment of the present application;

[0039] Figure 2 is a schematic diagram of the video structured system design engine framework in an example provided by the specific embodiment of the present application;

[0040] Figure 3is a data frame data structure schematic diagram in an embodiment provided by the specific embodiment of the present application;

[0041] Figure 4 is a schematic diagram of an action processing component library framework disclosed in an embodiment provided by the specific embodiment of the present application;

[0042] Figure 5 is a further refined schematic diagram of the matching bridge unit disclosed in an embodiment provided by the specific embodiment of the present application;

[0043] Figure 6 is a further preferred schematic diagram of an action processing component library framework disclosed in an embodiment provided by the specific embodiment of the present application;

[0044] Figure 7 is a schematic diagram of the further refined preferred selector provided by the specific embodiment of the present application;

[0045] Figure 8 is a further preferred schematic diagram of an action processing component library framework disclosed in an embodiment provided by the specific embodiment of the present application;

[0046] Figure 9 is a further preferred schematic diagram of an action processing component library framework disclosed in an embodiment provided by the specific embodiment of the present application;

[0047] Figure 10 is a further preferred schematic diagram of an action processing component library framework disclosed in an embodiment provided by the specific embodiment of the present application;

[0048] Figure 11 is a video structured system design flowchart disclosed in an embodiment provided by the specific embodiment of the present application;

[0049] Figure 12 is a further preferred video structured system design flowchart disclosed in an embodiment provided by the specific embodiment of the present application;

[0050] Figure 13 is a face snapshot structured application schematic diagram disclosed in an embodiment provided by the specific embodiment of the present application;

[0051] Figure 14 is a license plate recognition structured application schematic diagram disclosed in an embodiment provided by the specific embodiment of the present application.

[0052] Wherein, 1, action processing component library; 2, modeling module; 3, execution module; 11, action processing unit; 111, video source reading unit; 112, detector; 113, mapper; 114, filter; 115, matcher; 116, matching bridge unit; 117, preferencer; 1161, matching input bridge unit; 1162, matching output bridge unit; 1171, full cycle preferencer; 1172, selected cycle preferencer; 1173, minimum threshold preferencer; 1174, bubble preferencer; 118, frame divider; 119, saving unit; 11T, tracking unit; 111a, face detection unit; 114a, small face filtering unit; 115a, detection-tracking matching unit; 119a, figure saving unit; 117a, quality preferencer; 119b, snapshot result saving unit; 111b, multi-target detection unit; 114b, non-vehicle license plate filtering unit; 114c, small target filtering unit; 113a, vehicle identification mapping unit; 119c, identification result saving unit. DETAILED DESCRIPTION

[0053] In order to make the technical problems solved by the present application, technical solutions and beneficial effects clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0054] In the description of the present application, it should be understood that the terms "longitudinal", "radial", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0055] In the description of the present application, it should be noted that, unless otherwise specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0056] The video structured system design engine disclosed in the present application will be specifically explained in the present example in combination with the drawings, as follows Figure 1As shown, the video structured system design engine comprises the following modules:

[0057] An action processing component library 1, which comprises a plurality of action processing units 11; the action processing units 11 are used to load the processing results in the data frames after corresponding action processing on the input data frames, and then output the data frames;

[0058] A modeling module 2, which is used to select action processing units 11 from the action processing component library 1, and connect the selected action processing units 11 as nodes to form a behavior graph according to business requirements;

[0059] An execution module 3, which is used to parse and execute the behavior graph, and generate a target video structured system.

[0060] In order for those skilled in the art to further understand the present application, the names such as "video structured", "object" and "behavior graph" are introduced first.

[0061] The original video image (i.e. video source) is actually unstructured data, which cannot be directly read and recognized by a computer. In order to make the video image have better application in various application scenarios, intelligent video analysis technology must be used to structure the video image, which is video structured.

[0062] The object (English name: Object) in the present application refers to a physical object or target in video structured. In the art, common objects include face, head, human body, motor vehicle, non-motor vehicle, motor vehicle license plate, etc.

[0063] Video structured mainly has three types: the first type is the identification of objects (targets), that is, the identification of moving objects in the picture, for example, whether it is a person or a vehicle; the second type is the identification of object (target) features, that is, what features the moving person, vehicle or object in the picture has, for example, if it is a person, whether it is a man or a woman, whether it wears glasses, and what color clothes it wears; if it is a vehicle, what is the license plate number, what color is it, and what type of vehicle is it, etc.; the third type is the trajectory analysis of objects (targets), for example, analyzing whether the person or vehicle in the picture turns left or right or wanders, etc. In short, video structured is a kind of intelligent analysis, which can perform deep mining and information extraction on video data, etc. The intelligent analysis process usually adopts time-space segmentation, feature extraction, object recognition, deep learning and other processing means according to the semantic relationship of video content, and outputs text information or visualized graphical information that can be understood by computers and humans.

[0064] Regarding the modeling function of its modeling module 2, the method of connecting the action processing units 11 in modeling module 2 to form a behavior diagram is well known to those skilled in the art and can be implemented in the software development field without creative effort. For example, behavior diagrams can be implemented using UML (Unified Modeling Language), a graphical language that supports modeling and software system development, providing modeling and visualization support for all stages of software development, including from requirements analysis to specifications, construction, and configuration. Behavior diagrams are used to record changes in the interaction and instantaneous state of a model over time; and to track how the system behaves in a real environment, as well as to observe the system's reaction to an operation or event and its result. The behavior diagram in this application is a unidirectional acyclic graph; it can form multiple branches, which can be related but do not form loops.

[0065] In summary, the video structuring system design engine disclosed in this application is essentially a development and design software specifically designed for video structuring applications (or video structuring systems). As follows... Figure 2 As shown, the user interface of this development and design software can be similar to Microsoft Visio. For example, a motion processing component library 1 is set on the left side of the interface, and a modeling window is set in the middle area of ​​the interface. The function of the modeling module 2 is implemented in the modeling window, and an execution module 3 is set at the bottom of the modeling window. Developers can select the motion processing units 11 they need from the motion processing component library 1 on the left, drag them into the modeling window on the interface, and connect the motion processing units 11 to form a behavior graph of the corresponding business function. Finally, the execution module 3 can parse and execute the behavior graph to generate the target video structuring system.

[0066] The motion processing unit 11 is a collection of technical units derived from the abstraction of underlying video structuring technologies. These technical units are not limited to specific application scenarios but are only associated with the basic functions of the underlying technologies. This allows developers to implement various business functions (i.e., application scenarios) based on combinations of motion processing units 11.

[0067] In each action processing unit 11, signal interaction is typically achieved through data frames, which serve as the basic data structure, such as... Figure 3As shown, in the base data structure in this example, in addition to the header file, the data frame generally also includes data information. The data information at least includes image information, currently detected object information, object being tracked information, and object disappearing information. The image information can include but is not limited to RGB image, depth image, grayscale image, IR image, etc. Generally, in the process of video structuring, the objects that need to be focused on can be divided into three types according to their states: currently detected object, object being tracked, and object disappearing. Therefore, the data information related to the objects at least includes currently detected object information, object being tracked information, and object disappearing information. By pre-setting the data frame flowing through each node, a field is left for the processing result of the action processing unit 11 in each node, so that the processed result can be output to the subsequent node through the data frame. The processing result of each node can be written into the corresponding field of the data frame. In the data frame, the fields of image information, currently detected object information, object being tracked information, and object disappearing information can be set according to the protocol, so that the corresponding information can be written into the corresponding field. The data frame does not limit to only including the above information, but can also include other information. According to the need, the field of related information can be added to make the processing result in each action processing unit 11 be written into the corresponding field of the data frame. For example, a field of a specific length is used to store image information, a field of a specific length is used to store currently detected object information, object being tracked information, and object disappearing information, etc.

[0068] In the action processing unit 11, there are input, processing, and output processes. The input content is generally in the form of a data frame, which is processed internally and then the processing result is loaded into the data frame for output. Each action processing unit 11 can have multiple inputs and / or multiple outputs according to the need.

[0069] The correct parsing and execution of the behavior graph in the execution module 3 ensure the correct transmission of data between different action processing units 11. According to the prior setting, the flow of the entire behavior graph is optimized for multi-threading and heterogeneous computing. The output result obtains a video structuring system (i.e., a video structuring application).

[0070] The video structured system design engine disclosed in the application designs a plurality of action processing units 11 with minimum action processing functions by analyzing the business processes needed in the video structured design, the action processing units 11 are integrated in the action processing component library 1 for designers to select and combine according to business requirements; the modeling module 2 connects each action processing unit 11 as a node to form a behavior graph that can realize the business functions; the execution module 3 can correctly parse and execute the behavior graph, and finally generate a target video structured system. The video structured system design engine decouples all actions involved in the video structured design into independent action processing units 11, connects these action processing units 11 to form a behavior graph, so that a behavior graph can represent a complete video structured process, and the target video structured system that meets the design requirements can be obtained after execution. In this way, the tedious data structure management, heterogeneous computing, multi-thread synchronization and other transactions can be handled by the underlying execution engine. This measure enables the development designer to devote all his energy to the description of specific business, without having to worry about numerous underlying transactions, greatly simplifying the upper code of the video structured, reducing the development time and difficulty of the video structured application, and improving the development and debugging efficiency of the video structured application.

[0071] To enable those skilled in the art to understand, the specific implementation of the action processing unit 11 will be illustrated below by this example. As shown in the following figure, the action processing unit 11 includes: Figure 4

[0072] A video source reading unit 111 is used to read the video source that needs to be structured, and output data frames that can be processed by the subsequent action processing unit 11. The data frame at least contains image information of the video source, and reserves fields including current detected object information, object being tracked information and object being disappeared information. The video reading unit 111 is used as a general starting node.

[0073] ​The detector 112 is configured to detect image information contained in the input data frame, obtain current detected object information, and write the current detected object information into the data frame for output. The function of the detector 112 is to create and generate objects. After detecting the image information in the input data frame, the detector 112 identifies the features of the objects according to the content of the image information by using an intelligent model such as deep learning to implement the detection function. For example, the detector 112 can implement functions such as face detection, multi-target detection, motor vehicle detection, and non-motor vehicle detection, and finally output object information. For example, the detector 112 can be further divided into a human body detector, a multi-target detector, a motor vehicle detector, a non-motor vehicle detector, and the like. The various face detectors, multi-target detectors, and the like listed above can be further divided as needed, and thus will not be described again. Taking the face detector as an example, the detector 112 extracts an RGB image or a grayscale image from the data frame, detects face information from the image by using a deep learning algorithm, and stores the face information in the “current detected object information” field in the data frame. Taking the human body detector as an example, the detector 112 extracts an RGB image or a grayscale image from the data frame, detects human body information from the image by using a deep learning algorithm, and stores the human body information in the “current detected object information” field in the data frame. Taking the multi-target detector as an example, the detector 112 extracts an RGB image or a grayscale image from the data frame, detects object information of objects such as faces, human bodies, heads, motor vehicles, non-motor vehicles, and license plates from the image by using a deep learning algorithm, and stores the object information in the “current detected object information” field in the data frame. For example, when the detected target is a human, the object information includes a face frame, face frame position information, face feature information, face attribute information, and the like. When a motor vehicle is detected, the object information includes motor vehicle feature information and license plate position information. When a non-motor vehicle is detected, the object information includes non-motor vehicle feature information and non-motor vehicle detection frame information. The detector 112 can be further divided according to different deep learning algorithms.

[0074] The mapper 113 is used to map the current detected object information or the object being tracked information from the input data frame, generate a mapping result and write it into the data frame for output. The mapper 113 is essentially different from the detector 112. The mapper 113 operates on objects. It must be object information that has been detected. That is, it cannot generate objects from image information like the detector 112. Instead, it performs some mapping operation on objects. It can perform corresponding mapping operations on the current detected object information and the object being tracked information. It only changes the information of the object. For example, taking face information as an analysis target, the mapper 113 analyzes the age of the face and outputs the age field to fill in the data frame. For example, taking license plate information as an analysis target, the mapper 113 analyzes the license plate image obtained from the data frame and outputs the specific text content of the license plate to fill in the corresponding field. The mapper 113 does not increase or decrease any content. It only performs specific analysis on each input data frame to generate specific results and fill them into the corresponding fields. The following examples further illustrate the face age mapper: taking a series of face information as input, the mapper 113 predicts the age of each face through a deep learning algorithm and writes the age result into the age field of the face in the data frame; the face gender mapper: taking a series of face information as input, the mapper 113 predicts the gender of each face through a deep learning algorithm and writes the age result into the gender field of the face in the data frame; the license plate recognition mapper: taking a series of license plate position information as input, the mapper 113 predicts the content of each license plate through a deep learning algorithm and writes the result into the content field of each license plate in the frame information.

[0075] The filter 114 is used to filter a selection function with the selected object information as the independent variable and a logical value as the dependent variable from the input data frame and write the filtering result into the data frame for output. The result is only filtered from the input current data frame. The opposite description of filtering is filtering. When filtering the selected object as the independent variable, the output result of the filtering function based on the logical value as the dependent variable is true or false. If the output result is false, the selected object is filtered out, and thus the object information is deleted from the corresponding field in the data frame. If it is true, the selected object is retained, that is, the object information can be retained in the corresponding field in the data frame. In short, the filter 114 is usually responsible for filtering out the object being tracked information and the current detected object information that does not meet the requirements in the current data frame. There are 10 faces in the current image information, and some attributes of the faces need to be filtered out, that is, faces that do not meet the conditions are filtered out. Common filters 114 include quality filters and the like.

[0076] The matcher 115 is configured to receive a plurality of object sets from the input data frame and perform logical operation on the object sets to output a matched set after the logical operation. The object set includes a plurality of objects in the same data frame, such as a set of face, body, head, motor vehicle, non-motor vehicle, license plate, etc. The matched set outputted after the matching of the matcher can be a matched set between different object sets. The matched set can also include a set of objects in different data frames. Through the object sets in different historical frames, the matcher outputs a matched set of the following types: a detected object set, a being-tracked object set and a being-lost object set. For example, the matcher 115 receives object set A and object set B from two input nodes and matches the object set A and the object set B through a matching algorithm to divide them into three parts: (A-B) in the object set A only, (B-A) in the object set B only and a matched part (M) of the object set A and the object set B, which are respectively sent to three output nodes for use by subsequent task processing units. For example, the matcher 115 receives object set A, object set B and object set M' from three input nodes and matches the object set A and the object set B through a matching algorithm to divide them into three parts: (A-B) in the object set A only, (B-A) in the object set B only and a matched part (M) of the object set A and the object set B, which are respectively sent to three output nodes for use by subsequent processing units. The matcher 115 has many application scenarios. For example, when the video source is continuously input, the first frame has no tracked object and all the objects are detected objects. The matching result and the tracking result are matched. The newly appeared object is added as a tracked object. At this time, the tracked object, the lost object and the newly appeared object need to be analyzed. At this time, the matcher 115 can be used to distinguish the above objects. For another example, the multi-target detector 112 detects a head to obtain a body. However, the head and the body can have no corresponding relationship. At this time, the matcher 115 is needed to match. The head is assumed to be set A and the body is assumed to be set B. Through the matcher 115, the matched head and face can be outputted as a set and the rest of the unmatched head and face are outputted as another two sets. For another example, the multimedia matcher detects the corresponding relationship between the body and the non-motor vehicle. The matcher finally outputs the matched set in the field of the data frame.

[0077] The matching bridge unit 116 is configured to connect the matcher 115 with other nodes. As shown in FIG. 1, the matching bridge unit 116 includes a matching input bridge unit 1161 and a matching output bridge unit 1162. Figure 5

[0078] ​The matching input bridge unit 1161 is used to connect the matcher 115 with the action processing unit 11 of the previous input node, and the matching output bridge unit 1162 is used to connect the matcher 115 with the action processing unit 11 of the next output node.

[0079] It should be noted that the action processing unit 11 described above in this example is a basic functional unit that needs to be provided, and as a preferred mode, the action processing unit 11 can also be combined to generate a secondary development component with a specific business function; the secondary development component is also included in the action processing component library 1. In this way, the development and design process of the video structured system can be further simplified, the time can be shortened, and the efficiency can be improved.

[0080] Through the above-mentioned basic action processing unit 11, most of the design of the video structured application can be completed, and the application development and design needs of the developer can be met. Further reducing the development time and difficulty of the video structured application, improving the development efficiency and debugging efficiency of the video structured application.

[0081] In a further preferred embodiment, the technical solution in this example is mostly the same as the solution in the foregoing embodiments, the difference is that the action processing unit 11 in the action processing component library 1 is further optimized on the basis of embodiment 1. Among them, the action processing unit 11 further includes a preferencer 117, the preferencer 117 is used to select object information and corresponding data frames that meet the index requirements in a time period from a plurality of data frames input in the time period. The logic of the preferencer 117 is different from that of the above-mentioned checker and matcher 115, and the preferencer 117 mainly selects objects from the time perspective. That is, in the data frames from the time starting point a to the time ending point b, longitudinal comparison is performed to select from them. In other words, in a certain time period, the preferencer 117 considers the object image in the historical frame that meets the condition and the related index function of the object itself in history. According to the pre-prepared selection index function (this function takes an object as input and outputs an index value as an index) and the given selection strategy, the data frame of the object that meets the requirements including the optimal object and the corresponding return information in a certain time period is given. The selection index function can return the quality, area, etc. of the object.

[0082] As Figure 7 shown, the preferencer 117 is exemplified as follows, it should be noted that the preferencer 117 in this example is only preferred, and not only includes these preferencers 117. The preferencer 117 includes the following:

[0083] The full-cycle optimizer 1171 is used to select the data frame output corresponding to the best object information that meets the index requirements within the time period from the appearance to the disappearance of the target object; for example, after the face detector detects a face, it can filter out the face with the best quality and the corresponding data frame output from the image information from the beginning of the appearance of a specific face to the end of the disappearance of the face.

[0084] The period optimizer 1172 is used to select the data frame with the best performance in the past time period that meets the specified index requirements, using a period of several frames as the time period. For example, it can select the best performance frame from 50 frames as the time period. For instance, assuming a license plate is being photographed, 50 frames of image information are taken at the moment of exposure during the shooting process, and the clearest license plate and its corresponding data frame are selected and output.

[0085] The minimum threshold optimizer 1173 is used to select all frames that meet the specified criteria and whose criteria are better than the minimum threshold within a time period of several frames. This means that within a specific time period, as long as the minimum requirement of the relevant threshold is met, an output is selected. In this case, the output may not be unique, but rather multiple outputs are possible. The criteria of each output are recorded in the output data frame.

[0086] The bubble sorter 1174 is used to select data frames that meet the specified criteria and whose criteria are better than the previously selected optimal criteria within a time period of several frames. This means that within this time period, if a criterion in a data frame is better than the previously selected optimal criterion, the optimal criterion is output. Essentially, this method performs a bubble sort to optimize the data frames, and may ultimately output multiple criteria and corresponding data frames sequentially.

[0087] The addition of the Optimizer 117 allows for the rapid development of more systems that require optimization functionality in video structuring system design, without the need for additional manual coding.

[0088] In a further preferred embodiment, the technical solution in this example is largely the same as the solution in the above embodiments, the difference being that the motion processing unit 11 in the motion processing component library 1 is further optimized based on the above embodiments. For example... Figure 8 As shown, the action processing unit 11 also includes a frame divider 118, used to distribute data frames to processing branches formed by different nodes. For example, the frame divider 118 can store the current data frame on one hand, and on the other hand, distribute the data frame to other nodes in another branch for corresponding function processing on the other hand. With the addition of the frame divider 118, this can be implemented at any time.

[0089] In a further preferred embodiment, the technical solution in the present example is mostly the same as the solution in the above-mentioned embodiment, and the difference is that the action processing unit 11 in the action processing component library 1 is further optimized on the basis of the above-mentioned embodiment. As shown in Figure 9 The action processing unit 11 further includes a saving unit 119 for saving the data frames output in the previous node. The data frames saved in the saving unit 119 can also be called by other nodes, and the saving unit 119 can have multiple branches that are finally collected in the saving unit 119 for saving. In the same system, there can also be multiple saving units 119 to achieve the saving of different data frames.

[0090] In a further preferred embodiment, the technical solution in the present example is mostly the same as the solution in the above-mentioned embodiment, and the difference is that the action processing unit 11 in the action processing component library 1 is further optimized on the basis of the above-mentioned embodiment. As shown in Figure 10 The action processing unit 11 further includes a tracking unit 11T for tracking the to-be-tracked object in the input data frame to output the tracking result.

[0091] As a preferred mode, the action processing component library 1 further includes a secondary development component with a specific business function, which is generated by combining the action processing units 11.

[0092] The above-mentioned modules in the video structured system engine can be all or partially realized by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned modules.

[0093] In an embodiment, the video structured system design method disclosed by the present application will be specifically explained and described. The design method (or development method) in the present example includes developing and designing by using the video structured design engine provided in the above-mentioned embodiment; as shown in Figure 11 The video structured system design method includes the following steps:

[0094] S1, target business process setting step: receiving a process setting instruction, setting a business process of a target video structuring system according to the process setting instruction; for example, assuming that a face capture structuring system needs to be designed in the video structuring system design engine, the business process that needs to be included in face capture needs to be analyzed first, and each single basic node is obtained by analyzing it. For example, the first step needs to obtain the video source image captured by the camera, then track the video frame where the face appears, perform face detection on the corresponding video frame, obtain face detection information, filter the data frame where the face is detected, and match the results after detection and filtering, and optimize the output of the final output face quality. In this process, it also needs to consider which information needs to be saved through the saving unit 119. Based on the above analysis of the process, the corresponding action processing unit 11 can be selected in the next step to model the behavior graph.

[0095] S2, action processing unit 11 selection step: receiving an action selection instruction, selecting an action processing unit 11 associated with the business process according to the action selection instruction; in this process, according to the business process constructed in the previous step, the action processing unit 11 is selected as the basic node in the above business process, for example, still taking the implementation of the face capture structured video system as an example, in this process, the combination of the action processing unit 11 such as the video source reading unit 111, the tracking unit 11T, the frame divider 118, the detector 112, the filter 114, the matcher 115, the optimizer 117, and the saving unit 119 is placed in each node in the business process designed in the above step. When the action processing unit 11 corresponding to the business process is missing in the action processing component library 1, the development and design personnel can propose the demand of the related business process, and the maintenance personnel of the video structuring design engine can add the action processing unit 11 with corresponding functions. In this way, when the development and design personnel develop and design the target video structuring system, the maintenance personnel can add the action processing unit 11 with corresponding functions after the demand is proposed, further reducing the development time and difficulty of the video structuring application, and improving the development efficiency and debugging efficiency of the video structuring application.

[0096] S3, modeling step: modeling the selected corresponding action processing unit 11, i.e. structuring the business process of the target video according to the needs, connecting each action processing unit 11 associated with the business process as a node of the behavior graph; each action processing unit 11 is modeled in series to form a behavior graph known to the public, which can use a frame divider 118 to form multiple branches during the formation of the behavior graph. However, the behavior graph formed in the present application does not form a feedback loop. Similarly, taking the face capture structured system as an example, after connecting the action processing units 11 represented by each node in the modeling window in order, the behavior graph of the face capture structured system is obtained. No further description is given.

[0097] S4, execution step: parsing and executing the behavior graph to generate the target video structured system. In this process, ensure correct data transmission between different action processing units 11; correctly perform multi-threading and heterogeneous computing optimization on the entire process according to the prior settings. Through this step, the target video structured system is obtained, for example, the behavior graph of the face capture system formed by the above steps is obtained, and the face capture structured system is obtained after this step.

[0098] The video structured system design method provided in this example designs the video structured system in the video structured design engine disclosed in the present application. Without considering the needs and differences of the target video structured system, only the action processing units 11 with corresponding functions in the action processing component are connected to form a behavior graph, so that one behavior graph can represent a complete video structured process, and the target video structured system meeting the design requirements can be obtained after execution. In this way, the tedious data structure management, heterogeneous computing, multi-threading synchronization and other transactions can be handled by the underlying execution engine. This measure allows the developer to focus on the description of the specific business without worrying about numerous underlying transactions, greatly simplifying the upper code of the video structured system, reducing the development time and difficulty of the video structured application, and improving the development and debugging efficiency of the video structured application.

[0099] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0100] In an embodiment, the video structured system design method disclosed in the present application is specifically explained and described. The design method (or development method) in this example is mostly the same as the above embodiment, and as a preferred embodiment, a verification step is further added. As shown in Figure 12 The video structured system design method comprises the following steps:

[0101] S1. Target business process formulation steps: Receive process setting instructions, and set the business process of the target video structuring system according to the process setting instructions;

[0102] S2, Action processing unit 11 selection steps: Receive action selection instruction, and select the action processing unit 11 associated with the business process according to the action selection instruction;

[0103] S3. Modeling steps: Model the selected corresponding action processing unit 11, that is, according to the business process of the target video structuring system to be implemented, the action processing unit 11 associated with the business process is used as a node connection behavior graph.

[0104] S4. Execution steps: Parse and execute the behavior graph to generate the target video structured system.

[0105] S5. Verification Steps: Conduct a trial run of the generated target video structuring system to verify whether it meets the design requirements.

[0106] This example allows for further verification of the developed target video structuring system to confirm whether it meets the design requirements. It ensures that the target video structuring system operates correctly according to the defined logic.

[0107] In one embodiment, to enable those skilled in the art to further understand the present invention, the following example illustrates a target video structuring system obtained by using the video structuring design engine disclosed in the present invention for video structuring design.

[0108] like Figure 13 As shown, this example discloses a structured face capture application system, which includes the following action processing unit 11:

[0109] The video source reading unit 111 is used to read the original video source captured by the face capture camera and output a data frame containing graphic information.

[0110] The tracking unit 11T is used to track data frames containing human face images in graphic information.

[0111] Frame divider 118 is used to divide the data frame output by the tracking unit 11T into two branches, one of which is saved to the image storage unit 119a below, and the other is output to the face detection unit 111a of the next node.

[0112] The face detection unit 111a is used to detect face units appearing in the data frame and output the face detection results to fill the data frame.

[0113] The small face screening unit 114a screens out small faces from the detected faces and saves the qualified results in the data frame.

[0114] The matching input bridge unit 1161 bridges the small face screening unit 114a and the detection-tracking matching unit 115a of the next node.

[0115] The detection-tracking matching unit matches the results of face detection and tracking, outputs the matching results, and saves the matching results in the data frame.

[0116] The matching output bridge unit 1162 bridges the detection-tracking matching unit 115a and the graphic saving unit 119a of the next node.

[0117] The graphic saving unit 119a saves the data frame of the original video source and the data frame after face detection, small face screening, and detection-tracking matching.

[0118] The quality optimization unit 117a performs quality optimization on the data frame from the graphic saving unit 119a and outputs the best quality face image from the captured face images.

[0119] The capture result saving unit 119b saves the best quality face image obtained by capture as the capture result.

[0120] In an embodiment, to further enable those skilled in the art to understand the present application, the following will illustrate a target video structured system obtained by using the video structured design engine disclosed in the present application.

[0121] As shown in Figure 14 , the present embodiment discloses a license plate recognition application system. It includes the following action processing unit 11:

[0122] The video source reading unit 111 reads the original video source containing license plate information captured by a camera and outputs a data frame containing graphic information.

[0123] The tracking unit 11T tracks the data frame with license plate information in the graphic information.

[0124] The frame divider 118 divides the data frame output by the tracking unit 11T into two branches, one of which is saved by the graphic saving unit 119a below, and the other of which is output to the multi-target detection unit 111b of the next node.

[0125] The multi-target detection unit 111b detects multiple targets appearing in the data frame, outputs the multi-target detection results, and saves them in the data frame.

[0126] The non-license plate filtering unit 114b filters out the non-license plate targets in the multi-target result and only keeps the license plate in the data frame.

[0127] The small target filtering unit 114c filters out the small targets that do not meet the requirements in the license plate detection result and keeps the results that meet the requirements in the data frame.

[0128] The matching input bridging unit 1161 bridges the small target filtering unit 114c and the detection-tracking matching unit 115a of the next node.

[0129] The detection-tracking matching unit matches the results of license plate detection and tracking, outputs the matching result, and saves the matching result in the data frame.

[0130] The matching output bridging unit 1162 bridges the detection-tracking matching unit 115a and the figure saving unit 119a of the next node.

[0131] The figure saving unit 119a saves the data frame of the original video source and the data frame after the above multi-target detection, non-license plate filtering, small target filtering, and detection-tracking matching.

[0132] The quality optimization unit 117a performs quality optimization on the figure saving unit 119a to select the best quality license plate image from the captured license plate images and outputs the selected image.

[0133] The vehicle identification mapping unit 113a obtains the license plate image from the data frame, identifies the specific text content of the license plate after mapping analysis, and fills the corresponding field.

[0134] The identification result saving unit 119c saves the license plate information obtained after mapping analysis as the result.

[0135] In an embodiment, a computer device is provided, comprising a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, the computer readable instructions being the computer readable instructions generated by the video structured system design engine and the computer readable instructions generated by the video structured system design method. The computer device can be a server, the computer device comprising a processor, a memory, a network interface, and a database connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer readable instructions, and a database. The internal memory provides an environment for the operating system and the computer readable instructions in the readable storage medium to run. In an embodiment, one or more readable storage mediums are provided, the readable storage medium storing the codes involved in the video structured system design engine and the video structured system design method.

[0136] The computer readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium, and when executed, the computer readable instructions can implement the functions of the embodiments of the above devices. Any reference to memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above described functions.

[0138] The above merely describes preferred embodiments of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A video structuring system design engine, characterized in that, include: An action processing component library includes several action processing units. Each action processing unit performs corresponding action processing on an input data frame and then loads the processing result into the data frame for output. Each action processing unit includes a matcher, which receives several sets of objects from the input data frame, performs logical operations on the sets of objects, and outputs a matching set after logical operations. The object set includes multiple object sets in the same data frame. After the matcher matches multiple object sets in the same data frame, it outputs a matching set between different object sets. The object set also includes a set of objects in different data frames. After passing through the matcher, the set of objects in different data frames is output as: a set of detected objects, a set of objects being tracked, and a set of objects disappearing. The modeling module is used to select action processing units from the action processing component library and connect the selected action processing units as nodes to form a behavior graph according to business requirements. The execution module is used to parse and execute the behavior graph to generate the target video structuring system.

2. The video structuring system design engine according to claim 1, characterized in that, The data frame includes at least image information, information on currently detected objects, information on objects being tracked, and information on objects disappearing.

3. The video structuring system design engine according to claim 2, characterized in that, The motion processing unit in the motion processing component library includes: The video source reading unit is used to read the video source that needs to be structured and output data frames that can be processed by the subsequent motion processing unit. The detector is used to detect the image information contained in the input data frame, obtain the information of the currently detected object, and write the information of the currently detected object into the data frame for output. The mapper is used to map the information of the currently detected object or the object being tracked from the input data frame, and write the mapping result into the data frame for output. A filter is used to filter the input data frame using a filter function with object information as the independent variable and a logical value as the dependent variable, and writes the filtering results into the data frame for output. A matching bridge unit is used to connect the matcher to other nodes.

4. The video structuring system design engine according to claim 3, characterized in that, The matching bridging unit includes a matching input bridging unit and a matching output bridging unit; The matching input bridging unit is used to connect the matcher to the action processing unit of its preceding input node, and the matching output bridging unit is used to connect the matcher to the action processing unit of its following output node.

5. The video structuring system design engine according to claim 3, characterized in that, The motion processing unit also includes an optimizer, which is used to select object information that meets the index requirements and output the corresponding data frames from a number of data frames input within a certain time period.

6. The video structuring system design engine according to claim 5, characterized in that, The preferred device includes the following: The full-cycle optimizer is used to select the data frame output corresponding to the optimal object information that meets the above-mentioned index requirements within the time period from the appearance to the disappearance of the target object. A period optimizer is selected to output the data frame that meets the index requirements within a past time period, using a number of frames as the time period. The minimum threshold optimizer is used to select all frames that meet the index requirements and whose indexes are better than the minimum threshold, and output them, with a time period of several frames. A bubble sorter is used to select data frames that meet the specified criteria and whose criteria are better than the previously selected optimal criteria within a time period of several frames.

7. The video structuring system design engine according to claim 5, characterized in that, The action processing unit also includes a frame splitter, which is used to distribute data frames to different processing branches formed by nodes.

8. The video structuring system design engine according to claim 5, characterized in that, The action processing unit also includes a storage unit for saving the data frames output from the preceding nodes.

9. The video structuring system design engine according to claim 5, characterized in that, The motion processing unit also includes a tracking unit for tracking the object to be tracked in the input data frame and outputting the tracking result.

10. The video structuring system design engine according to claim 3, characterized in that, The motion processing component library also includes secondary development components with specific business functions generated by combining the motion processing units.

11. A video structuring system design method, characterized in that, The video structuring system design method utilizes the video structuring system design engine described in any one of claims 1-10 to design a video structuring system; the video structuring system design method includes the following steps: Receive process setting instructions and set the business process of the target video structuring system according to the process setting instructions; Receive an action selection instruction, and select an action processing unit associated with the business process according to the action selection instruction; Each action processing unit associated with the business process is connected as a node in the behavior graph; The behavior graph is parsed and executed to generate the target video structuring system.

12. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, The computer-readable instructions are the readable instructions generated by the video structuring system design engine according to any one of claims 1-10 and the readable instructions generated by the video structuring system design method according to claim 11.

13. One or more readable storage media storing code related to the video structuring system design engine and video structuring system design method as described in any one of 1-11.

Images