A multi-path video stream target identification method based on an edge computing device

Abstract

The application relates to a multi-path video stream target recognition method based on an edge computing device, multi-thread access to multi-path video streams, multi-process parallel acceleration target recognition for multiple target detection models, flexible self-defined comprehensive judgment of the obtained recognition results according to user requirements, real-time analysis of the video streams, and corresponding alarm analysis data and real-time recognition video streams are provided.

Description

TECHNICAL FIELD

[0001] The application relates to a multi-path video stream target recognition method based on an edge computing device and belongs to the technical field of computer vision. BACKGROUND

[0002] In recent years, with the rapid development of edge computing and artificial intelligence technology, domestic enterprises have invested a large amount of resources and efforts in the research and development of edge computing devices and AI video analysis products. The edge computing device is a computing device that places computing power at the edge of the network, and has the characteristics of real-time response, low delay and large capacity, meeting the needs of various application scenarios. The research and development of AI video analysis products is also one of the key areas of current domestic enterprises. AI video analysis technology is based on deep learning and computer vision algorithms, which can intelligently analyze and process video data. Through the optimization of algorithms such as recognition, detection and tracking of video content, AI video analysis products can be widely used in video surveillance, intelligent transportation, intelligent manufacturing and other fields. Domestic enterprises have made a lot of investment in algorithm research and development, hardware devices and software platforms, and gradually realized the localization of AI video analysis products.

[0003] The prior art such as the invention patent with the application number "202311512133.2" discloses an edge computing method and device for machine vision and a storage medium. The visual data is collected by a visual device and transmitted to an edge device through a network. The visual data is analyzed and processed in real time by a machine vision model on the edge device, and corresponding decision, control instruction and event trigger signal are generated. However, this scheme only uses a single model built into the edge device for inference, and the code of the model itself needs to be recompiled for different recognition targets, which cannot provide users with custom settings and has relatively low practicality. When dealing with multiple visual devices, parallel inference cannot be performed, and the processing efficiency needs to be improved. SUMMARY

[0004] In order to solve the problems existing in the prior art, the application provides a multi-path video stream target recognition method based on an edge computing device.

[0005] The technical scheme of the application is as follows:

[0006] On the one hand, the application provides a multi-path video stream target recognition method based on an edge computing device, which includes the following steps:

[0007] A model library is constructed, the model library is used to store pre-trained target recognition models, each target recognition model is provided with a limited number of recognition labels, and each recognition label corresponds to a type of recognition target;

[0008] The method comprises the following steps: creating multiple processes, selecting one or more target recognition models in a model library, selecting corresponding recognition labels in each selected target recognition model according to the type of target to be recognized, loading a target recognition model in each process, and receiving multiple video streams by the edge computing device, decoding the video stream to obtain multiple frames of images, pre-processing the multiple frames of images, inputting the multiple frames of images into each process, and outputting real-time target recognition results of each target recognition model for the video stream after parallel inference of the target recognition models in all processes.

[0009] The final real-time target recognition result of the video stream is obtained by synthesizing the real-time target recognition result of each target recognition model.

[0010] As a preferred embodiment of the present application, an over-limit detection is arranged to detect the real-time target recognition result of each target recognition model.

[0011] The specific steps of the over-limit detection are as follows: a recognition target quantity threshold is preset, and an alarm is triggered when the recognition target quantity of the real-time target recognition result corresponding to any recognition label in the target recognition model is greater than the preset recognition target quantity threshold.

[0012] As a preferred embodiment of the present application, a low-limit detection is arranged to detect the real-time target recognition result of each target recognition model.

[0013] The specific steps of the low-limit detection are as follows: a recognition target quantity threshold is preset, and an alarm is triggered when the recognition target quantity of the real-time target recognition result corresponding to any recognition label in the target recognition model is less than the preset recognition target quantity threshold.

[0014] As a preferred embodiment of the present application, a cross detection is arranged to detect the real-time target recognition result of each target recognition model.

[0015] The specific steps of the cross detection are as follows: a cross area proportion threshold is set, and an alarm is triggered when there is a cross region in the real-time target recognition result corresponding to any two recognition labels of the target recognition model in a plane space, and the ratio of the cross region area to the non-cross region area is greater than the cross area proportion threshold.

[0016] As a preferred embodiment of the present application, the final real-time target recognition result of the video stream is converted into a video file of visualized target recognition result and stored in the edge computing device.

[0017] As a preferred embodiment of the present application, the final real-time target recognition result of the video stream is converted into a video stream of visualized target recognition result and pushed into a designated RTSP server in real time through an RTSP protocol.

[0018] As a preferred embodiment of the present application, the edge computing device pushes the video stream of the target recognition result visualization by using the built-in hardware codec function when pushing the video stream.

[0019] In another aspect, the present application also provides an edge computing device-based multi-channel video stream target recognition system, comprising a video stream input module, a model library construction module, a custom model module, a model inference module and a recognition result output module.

[0020] The video stream input module is configured to receive multi-channel video streams through video stream addresses.

[0021] The model library construction module is configured to construct a model library and add pre-trained target recognition models to the model library, each target recognition model being provided with a plurality of recognition labels, each recognition label corresponding to a type of recognition target.

[0022] The custom model module is configured to select one or more target recognition models in the model library and select corresponding recognition labels in the target recognition models according to the types of targets to be recognized by the video stream.

[0023] The model inference module is configured to create a plurality of processes, load a target recognition model selected by the custom model module in each process, and after the edge computing device receives multi-channel video streams, for any one video stream, first decode the video stream to obtain a plurality of frames of images, then pre-process the plurality of frames of images and input them into each process, and after the target recognition models in all processes are inferred in parallel, output the real-time target recognition result of each target recognition model for the video stream.

[0024] The recognition result output module is configured to synthesize the real-time target recognition result of each target recognition model for the video stream to obtain the final real-time target recognition result of the video stream, and output the video stream of the target recognition result visualization after conversion.

[0025] In another aspect, the present application also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of the embodiments of the present application when executing the program.

[0026] In another aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, wherein the program is executable by a processor to implement the method according to any one of the embodiments of the present application.

[0027] The present application has the following advantages:

[0028] 1、The present application combines multi-threading and multi-process to perform multi-model parallel inference on multi-channel video streams, and realizes flexible user-level detection target customization to quickly realize real-time analysis operation on end-to-end multi-channel video streams, and provides a good solution for the rapid landing of AI video analysis. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a method flowchart of the present application;

[0030] Figure 2 is an AI skill configuration flowchart of an embodiment of the present application;

[0031] Figure 3 is a target recognition visualization schematic diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0033] It should be understood that the step numbers used herein are only for the convenience of description, and are not limited to the execution sequence of the steps.

[0034] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clear from the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0035] The terms "include" and "contain" indicate the presence of the described features, whole, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0036] The term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0037] Embodiment one:

[0038] Referring to Figure 1 A multi-channel video stream target recognition method based on an edge computing device, comprising the following steps:

[0039] construct a model library for storing pre-trained target recognition models, each target recognition model being provided with a limited number of recognition labels, each recognition label corresponding to a type of recognition target;

[0040] create multiple processes, select one or more target recognition models in the model library and select corresponding recognition labels in each selected target recognition model according to the type of target to be recognized, load one target recognition model in each process, and after the edge computing device receives multiple video streams, for any video stream, first decode the video stream to obtain multiple frames of images, then pre-process the multiple frames of images and input them into each process, and after the target recognition models in all processes perform parallel inference, output the real-time target recognition result of each target recognition model for the video stream;

[0041] integrate the real-time target recognition result of each target recognition model for the video stream to obtain the final real-time target recognition result of the video stream.

[0042] As a preferred embodiment of the present embodiment, an over-limit detection is provided to detect the real-time target recognition result of each target recognition model;

[0043] The specific steps of the over-limit detection are: a recognition target quantity threshold is preset, and an alarm is triggered when the number of recognition targets in the real-time target recognition result corresponding to any recognition label in the target recognition model is greater than the preset recognition target quantity threshold;

[0044] For example, assuming that the recognition label is head and the recognition target quantity threshold is 5, when the model detects more than 5 "head" recognition results in the video stream, an alarm is triggered, and an alarm message is sent outward according to the actual push frequency.

[0045] As a preferred embodiment of the present embodiment, a low-limit detection is provided to detect the real-time target recognition result of each target recognition model;

[0046] The specific steps of the low-limit detection are: a recognition target quantity threshold is preset, and an alarm is triggered when the number of recognition targets in the real-time target recognition result corresponding to any recognition label in the target recognition model is less than the preset recognition target quantity threshold;

[0047] For example, assuming that the recognition label is waistcoat and the recognition target quantity threshold is set to 1, when the model detects less than 1 "waistcoat" recognition result in the video stream, an alarm is triggered, and an alarm message is sent outward according to the actual push frequency.

[0048] As a preferred embodiment of the present embodiment, a cross-detection is provided to detect the real-time target recognition result of each target recognition model;

[0049] The specific step of the cross detection is: setting a cross area proportion threshold, when there is a cross region on a plane space in real-time target recognition results corresponding to any two identification labels of a target recognition model, if the ratio of the cross region area to the non-cross region area is greater than the cross area proportion threshold, an alarm is triggered;

[0050] For example, in a certain target recognition model, two identification labels of wrongglove and operatingbar are selected to perform target recognition on a video stream, and the cross area proportion threshold is set to 0.01. When “not wearing gloves” and “operating bar” appear simultaneously in the video stream, and the window of the identification results corresponding to the two identification labels exists cross, and the ratio of the cross region area to the non-cross region area exceeds 0.01, an alarm is triggered, and an alarm message is sent according to the actual push frequency.

[0051] As a preferred embodiment of the present embodiment, the final real-time target recognition result of the video stream is converted into a target recognition result visualized video file stored in an edge computing device.

[0052] As a preferred embodiment of the present embodiment, the final real-time target recognition result of the video stream is converted into a target recognition result visualized video stream and pushed to a specified RTSP server in real time through an RTSP protocol, and the RTSP server is used to realize remote real-time viewing of the video stream.

[0053] As a preferred embodiment of the present embodiment, when the edge computing device pushes the target recognition result visualized video stream, the built-in hardware encoding and decoding function of the edge computing device is used to improve the pushing speed and quality of the video stream.

[0054] In the present embodiment, the target recognition model in the model library is a yolox model trained by the user himself, and each target recognition model in the model library can only recognize a limited number of target types. The user needs to select one or more different target recognition models according to the target types to be recognized;

[0055] The pre-training step of the yolox model is:

[0056] A training sample set is constructed, which includes image samples of targets to be recognized (image samples of helmets) and labels corresponding to the types of each image sample (the label is a helmet);

[0057] The training sample set is input into the yolox model, and the algorithm automatically adjusts the yolox model parameters during the training process until the model recognition accuracy reaches a preset threshold, and the training of the yolox model is completed.

[0058] The specific steps of the trained yolox model to recognize targets are:

[0059] The video stream is decoded in real time to obtain a series of multiple frames of images. Each frame of image is preprocessed (by scaling, padding, numerical standardization, and other uniform operations). The preprocessed frame of image is then input into the trained YOLOx model to obtain the corresponding target and its coordinate data in each frame of image.

[0060] After visualizing the target in each frame by using the target coordinate data of each frame (i.e., marking the target location with a rectangle and identifying the target type with text in each frame), all images are re-encoded and output as a video stream of the target visualization.

[0061] When multiple models are selected for multi-process recognition of a video stream, the target visualization results of each frame after decoding are merged from the different models. That is, each frame contains the recognition results of all selected models. Finally, all merged images are encoded and output as a video stream of target visualization. This target visualization video stream is as follows: Figure 3 As shown, targets are identified by rectangular boxes, and the name of the target type is displayed in the upper left corner of the rectangle.

[0062] In this embodiment, the multi-process implementation is based on Python, and its specific steps are as follows:

[0063] Import the required libraries;

[0064] Define a model loading function to load the target recognition model and return the model object;

[0065] Define a preprocessing function to preprocess the input image, such as scaling, cropping, normalizing, etc., and return the preprocessed image;

[0066] Define a post-processing function to process the output, such as drawing borders and labels, and return an image visualizing the detection results;

[0067] Define a model inference function to perform inference of a target recognition model in a process. The function takes a model object, an image path and an output path as parameters, and calls the preprocessing function and the postprocessing function in the function.

[0068] Create a process pool, specifying the number of processes and an initialization function. The initialization function can be used to load a target recognition model in each process.

[0069] Create a task list, each task contains an input image path and an image output path, and then pass the task list to the map method of the process pool, let each process execute a task;

[0070] Finally, wait for all processes to complete, close the process pool and release resources.

[0071] Example two:

[0072] A multi-channel video stream target recognition system based on an edge computing device, comprising a video stream input module, a model library construction module, a custom model module, a model inference module, and a recognition result output module;

[0073] The video stream input module is configured to receive multi-channel video streams through video stream addresses;

[0074] The model library construction module is configured to construct a model library and add pre-trained target recognition models to the model library, each target recognition model being provided with multiple recognition labels, each recognition label corresponding to a type of recognition target;

[0075] The custom model module is configured to select one or more target recognition models in the model library and select corresponding recognition labels in the target recognition models according to the types of targets to be recognized by the video stream;

[0076] The model inference module is configured to create multiple processes, and load a target recognition model selected by the custom model module in each process. After the edge computing device receives the multi-channel video stream, for any video stream, the video stream is first decoded to obtain multiple frames of images, and then the multiple frames of images are preprocessed and input into each process. After the target recognition models in all processes are inferred in parallel, the real-time target recognition results of each target recognition model for the video stream are output;

[0077] The recognition result output module is configured to synthesize the real-time target recognition results of each target recognition model for the video stream to obtain the final real-time target recognition result of the video stream, and convert the video stream into a visual video stream of the target recognition result and output.

[0078] In this embodiment, the system is built into an edge computing device and implemented by python code, and has a graphical interface;

[0079] The corresponding video stream url address is manually set through the graphical interface, the upper limit of the video stream setting is determined according to the performance of the edge device, and the system will start a corresponding number of threads according to the number of video streams;

[0080] The custom model module is embodied as an AI skill management interface in a graphical interface, in which existing AI skills can be turned on, turned off, modified, and deleted, and each AI skill can be customized by a user, see Figure 2 When the add skill button is clicked, the page displays an add AI skill window, through which the AI skill number and name are set, and multiple models can be added for the AI skill, after the model is selected in the AI skill window, a model configuration window is popped up, through which the model type to be added is selected and the recognition label of the model is selected, and the NMS threshold, target frame threshold, and input picture size can be configured, after the configuration is completed, a model is successfully configured by clicking save, for the case of different model recognition labels, other models can be added to select the corresponding recognition label, after the model configuration is completed, the skill warning category is selected through the skill warning category option, the threshold size is input in the detection object quantity threshold, and the message to be pushed out is input in the alarm information input box, and then uploaded, completing the configuration of the AI skill;

[0081] In the same scene, multiple video stream url addresses (i.e. multiple cameras) can be selected, and multiple AI skills can be selected for each camera, after the setting is completed, the task can be started, and the program will encode the visual detection result into an RTSP stream according to the requirements and push it to the specified RTSP server, and also push the analysis result to the message queue of the redis middleware, for the user to view and consume.

[0082] Embodiment three

[0083] The embodiment provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the program to realize the method in any embodiment of the application.

[0084] Embodiment four

[0085] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the method in any embodiment of the application.

[0086] In the embodiments of the present application, “at least one” means one or more, and “multiple” means two or more. “And / or” describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent the cases of A alone, A and B together, and B alone. Wherein A and B can be singular or plural. The character “ / ” generally represents an “or” relationship between the front and rear associated objects. “At least one of the following” and the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, wherein a, b, and c can be single or multiple.

[0087] Those of ordinary skill in the art can realize that the units and algorithm steps described in the embodiments disclosed herein can be realized in electronic hardware, computer software, and a combination of electronic hardware and computer software. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0088] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0089] In several embodiments provided in the present application, any function realized in the form of a software function unit and sold or used as an independent product can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory; hereinafter referred to as: ROM), a random access memory (Random Access Memory; hereinafter referred to as: RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0090] The above is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation based on the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for target recognition of multi-path video stream based on edge computing device, characterized in that, The method comprises the following steps: a model library is constructed for storing pre-trained target recognition models, each target recognition model is provided with a limited number of recognition labels, and each recognition label corresponds to a type of recognition target; a plurality of processes are created, one or more target recognition models are selected in the model library, corresponding recognition labels are selected in each selected target recognition model according to the type of target to be recognized, and then a target recognition model is loaded in each process; after the edge computing device receives a plurality of video streams, for any video stream, the video stream is first decoded to obtain a plurality of frames of images, the plurality of frames of images are preprocessed and then input into each process, and the target recognition models in all processes are inferred in parallel to output real-time target recognition results of each target recognition model on the video stream; the real-time target recognition results of each target recognition model on the video stream are integrated to obtain a final real-time target recognition result of the video stream; a target recognition model skill management graphical interface is constructed, and the target recognition model is configured through the target recognition model skill management graphical interface; in the target recognition model skill management graphical interface, an AI skill window is displayed by clicking an add skill button, and an AI skill number and a name are set in the window; in the AI skill window, a model is selected by clicking, a model configuration window is entered, a target recognition model type is selected and the recognition label of the target recognition model is specified, and a non-maximum suppression threshold, a target frame threshold and an input picture size are configured, so as to complete the configuration of the target recognition model; after the configuration of the target recognition model is completed, a target alarm category is selected through a skill alarm category option, a threshold size is input in a detection object quantity threshold input box, and an alarm message to be pushed out is input in an alarm information input box, so as to complete the configuration of the AI skill. 2.The method of claim 1, wherein, an over-limit detection is set to detect the real-time target recognition result of each target recognition model; the specific steps of the over-limit detection are as follows: a preset recognition target quantity threshold is set, and when the number of recognition targets in the real-time target recognition result corresponding to any recognition label in the target recognition model is greater than the preset recognition target quantity threshold, an alarm is triggered. 3.The method of claim 1, wherein, a low-limit detection is set to detect the real-time target recognition result of each target recognition model; the specific steps of the low-limit detection are as follows: a preset recognition target quantity threshold is set, and when the number of recognition targets in the real-time target recognition result corresponding to any recognition label in the target recognition model is less than the preset recognition target quantity threshold, an alarm is triggered. 4.The method of claim 1, wherein, a cross detection is set to detect the real-time target recognition result of each target recognition model; the specific steps of the cross detection are as follows: a cross area proportion threshold is set, and when there is a cross region in the real-time target recognition result corresponding to any two recognition labels of the target recognition model in the plane space, if the ratio of the area of the cross region to the area of the non-cross region is greater than the cross area proportion threshold, an alarm is triggered. 5.The method of claim 1, wherein, the final real-time target recognition result of the video stream is converted into a target recognition result visualized video file and stored in the edge computing device. 6.The method of claim 1, wherein, The final real-time target recognition result of the video stream is converted into a target recognition result visualized video stream and pushed into a designated RTSP server in real time through an RTSP protocol.

7. The method of claim 6, wherein the method further comprises: The edge computing device pushes the target recognition result visualized video stream through the built-in hardware coding and decoding function.

8. An edge computing device based multi-path video stream target identification system, comprising: The system comprises a video stream input module, a model library construction module, a custom model module, a model inference module, and an identification result output module. The video stream input module is configured to receive multiple video streams through video stream addresses. The model library construction module is configured to construct a model library and add pre-trained target recognition models to the model library, each target recognition model being provided with multiple identification labels, each identification label corresponding to a type of identification target. The custom model module is configured to select one or more target recognition models in the model library and select corresponding identification labels in the target recognition models according to the types of identification targets required by the video stream, specifically as follows: A target recognition model skill management graphical interface is constructed, and the target recognition model is configured through the target recognition model skill management graphical interface. In the target recognition model skill management graphical interface, an AI skill window is displayed by clicking the add skill button, and the AI skill number and name are set in the window. In the AI skill window, a model configuration window is entered by clicking the selected model, the target recognition model type is selected in the window, the identification label of the target recognition model is specified, and the non-maximum suppression threshold, target frame threshold, and input picture size are configured, thereby completing the configuration of the target recognition model. After the configuration of the target recognition model is completed, the target alarm category is selected through the skill alarm category option, the threshold size is input in the detection object quantity threshold input box, and the alarm message to be pushed out is input in the alarm information input box, thereby completing the configuration of the AI skill. The model inference module is configured to create multiple processes, load a target recognition model selected through the custom model module in each process, and after the edge computing device receives multiple video streams, for any video stream, first decode the video stream to obtain multiple images, pre-process the multiple images, and then input each image into each process, the target recognition models in all processes are inferred in parallel, and the real-time target recognition result of each target recognition model for the video stream is output. The identification result output module is configured to integrate the real-time target recognition result of each target recognition model for the video stream, obtain the final real-time target recognition result of the video stream, and output the target recognition result visualized video stream after conversion.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 to 7.

Images