Fireworks detection system based on C / S architecture and fireworks detection method, equipment and medium

Abstract

The present application discloses a fireworks detection system and detection method, equipment, and medium. The fireworks detection method includes shooting a fireworks detection video; reading frame images in the fireworks detection video to obtain images to be detected; parsing the fireworks detection training model to obtain the model name and image format requirements; preprocessing the images to be detected according to the image format requirements to obtain images to be inferred; creating model input data according to the images to be inferred; predefining the model output target set; sending the model input data, the model output target set, and the result callback function to the fireworks detection training model; the fireworks detection training model detects the model input data based on the model output target set to obtain the image inference result; filtering the image inference result according to the preset threshold, and packaging the filtered image inference result into serialized fireworks data and reporting it to the alarm platform. The present method can accurately, timely, and effectively meet the requirements for fireworks detection and has strong environmental adaptability.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent fire and smoke detection, and in particular to a fire and smoke detection system based on a C / S architecture and a fire and smoke detection method, device, and medium. Background Art

[0002] With the rapid development of social economy, the process of intelligence is accelerating, and highly intelligent workshops, factories, and parks are constantly emerging. The continuous intelligence of equipment also brings about dangerous factors of flammability and explosion, which brings new challenges to indoor and outdoor fire protection. How to quickly and accurately determine whether a fire has occurred, so as to deal with it in time and ensure the safety of property and life is an important issue in fire protection work.

[0003] Currently, the commonly used fire detection methods include: 1. Sensor detection method: using fireworks sensors to detect the gas concentration during fireworks, thereby detecting the occurrence of fireworks; 2. Video analysis method: performing color analysis on the video taken by the visible light camera, and adding a dynamic detection algorithm to determine whether fireworks are occurring.

[0004] The disadvantages of the above-mentioned fireworks detection methods are as follows: 1. Sensor detection method: There are many forms of sensors that can detect fireworks, but they all have the disadvantages of limited application space, susceptibility to interference, high false alarm rate, low intelligence, and unsuitable for detection under harsh conditions. 2. Video analysis method: Simply analyzing through video, judging by color and the dynamics of fireworks, there is a certain false alarm rate. When the distance of fireworks is relatively far, it is impossible to judge because the pixels of fireworks on the video are relatively small, resulting in missed alarms.

[0005] Therefore, there is an urgent need to develop a new type of smoke and fire detection system based on a C / S architecture, as well as a smoke and fire detection method, equipment, and medium based on the above-mentioned smoke and fire detection system to solve the above-mentioned problems. Summary of the invention

[0006] The embodiments of this specification provide a fire and smoke detection system based on a C / S architecture, as well as a fire and smoke detection method, device and medium based on the fire and smoke detection system, which are used to solve the following technical problems in the prior art: the existing fire and smoke detection methods have problems such as missed alarms, high false alarm rates, limited applicable space, blind spots in monitoring and poor environmental adaptability.

[0007] The embodiments of this specification adopt the following technical solutions:

[0008] A smoke and fire detection system based on a C / S architecture, wherein the system comprises:

[0009] A cloud server, the cloud server comprising at least a graphics processor and a gRPC port, wherein the graphics processor is provided with at least a smoke and fire detection training model, and the smoke and fire detection training model is provided in the graphics processor using an Nvidia triton architecture;

[0010] A client, the client at least comprising a video capture device and an intelligent inspection device, wherein the video capture device is arranged in the intelligent inspection device, the video capture device is at least used to shoot a smoke and fire detection video, and the intelligent inspection device is at least used to load the video capture device and drive the video capture device to perform inspection;

[0011] The cloud processor communicates with the client through the gRPC port, and the client accesses the smoke and fire detection training model through the gRPC service.

[0012] Furthermore, the client further includes:

[0013] An image reading module, the image reading module is used to obtain the video stream address of the fireworks detection video, obtain the fireworks detection video according to the video stream address using a real-time streaming protocol, and use the VideoCapture class of OpenCV to read at least one frame of the fireworks detection video to obtain a picture to be detected;

[0014] An image preprocessing module, the image preprocessing module is used to parse the smoke and fire detection training model to obtain the model name and image format requirements of the smoke and fire detection training model, and to preprocess the image to be detected according to the image format requirements to obtain the image to be inferred;

[0015] A model input module, the model input module is used to create model input data of a preset type according to the picture to be inferred;

[0016] A model output module, wherein the model output module is used to predefine a model output target set;

[0017] A model reasoning module, the model reasoning module is used to access the graphics processor through the gRPC service, and send the model input data, the model output target set and the result callback function to the smoke and fire detection training model;

[0018] The inference result post-processing module is used to filter the image inference results according to a preset threshold value, and package the filtered image inference results into serialized fireworks data using the rapidjson tool and report them to the alarm platform.

[0019] Furthermore, the preset type is a Nvidia trition-client type.

[0020] A method for detecting smoke and fire based on the smoke and fire detection system based on a C / S architecture according to any one of the above, wherein the method comprises:

[0021] Using the video capture device to shoot a smoke and fire detection video;

[0022] The client obtains the video stream address of the fire and smoke detection video, obtains the fire and smoke detection video according to the video stream address by using a real-time streaming protocol, and reads at least one frame of the fire and smoke detection video by using the VideoCapture class of OpenCV to obtain a picture to be detected;

[0023] The client accesses the smoke and fire detection training model through the gRPC service and parses the smoke and fire detection training model to obtain the model name and image format requirements of the smoke and fire detection training model;

[0024] The client preprocesses the image to be detected according to the image format requirement to obtain the image to be inferred;

[0025] The client creates model input data of a preset type according to the image to be inferred;

[0026] The client predefines a model output target set;

[0027] The client accesses the graphics processor through the gRPC service, and sends the model input data, the model output target set and the result callback function to the smoke and fire detection training model;

[0028] The smoke and fire detection training model detects the model input data based on the model output target set to obtain an image inference result, and sends the image inference result to the client according to the result callback function;

[0029] The client filters the image reasoning results according to a preset threshold, and packages the filtered image reasoning results into serialized fireworks data using the rapidjson tool and reports them to the alarm platform.

[0030] Furthermore, the image to be detected is in mat format.

[0031] Furthermore, the image format requirement includes at least one of the following: the length of the image, the width of the image, the color of the image, and the number of channels of the image.

[0032] Furthermore, the preprocessing of the image to be detected according to the image format requirement at least includes the following steps:

[0033] Use the cvColor function of OpenCV to perform color conversion on the image to be detected;

[0034] Use the resize function to scale the image to be detected to a size that meets the requirements of the image format;

[0035] The preprocessed image to be inferred is copied into the pre-specified cache variable through memcpy.

[0036] Furthermore, the model output target set includes at least one of the following target items: the number of fireworks ignition points, fireworks coordinates, fireworks types, and fireworks level assessment scores.

[0037] Furthermore, the smoke and fire detection training model detects the model input data based on the model output target set to obtain an image inference result, which at least includes the following steps:

[0038] The smoke and fire detection training model detects the model input data item by item according to the target items in the model output target set,

[0039] If the model input data contains data that meets the target item, the target item is marked according to a preset value in the image reasoning result;

[0040] If the model input data does not contain data that meets the target item, the target item is marked as empty in the image reasoning result.

[0041] Furthermore, the smoke and fire detection method further comprises:

[0042] In the Linux environment, the client is compiled into a .so dynamic link library, and the six model processing interfaces of the client are encapsulated into three C interfaces of client initialization, model reasoning and resource release for calling by the Go language to realize alarm reporting.

[0043] A smoke and fire detection device, wherein the device comprises:

[0044] at least one processor; and,

[0045] a memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0047] Using the video capture device to shoot a smoke and fire detection video;

[0048] The client obtains the video stream address of the fire and smoke detection video, obtains the fire and smoke detection video according to the video stream address by using a real-time streaming protocol, and reads at least one frame of the fire and smoke detection video by using the VideoCapture class of OpenCV to obtain a picture to be detected;

[0049] The client accesses the smoke and fire detection training model through the gRPC service and parses the smoke and fire detection training model to obtain the model name and image format requirements of the smoke and fire detection training model;

[0050] The client preprocesses the image to be detected according to the image format requirement to obtain the image to be inferred;

[0051] The client creates model input data of a preset type according to the image to be inferred;

[0052] The client predefines a model output target set;

[0053] The client accesses the graphics processor through the gRPC service, and sends the model input data, the model output target set and the result callback function to the smoke and fire detection training model;

[0054] The smoke and fire detection training model detects the model input data based on the model output target set to obtain an image inference result, and sends the image inference result to the client according to the result callback function;

[0055] The client filters the image reasoning results according to a preset threshold, and packages the filtered image reasoning results into serialized fireworks data using the rapidjson tool and reports them to the alarm platform.

[0056] A non-volatile computer storage medium for smoke and fire detection stores computer executable instructions, wherein the computer executable instructions are configured as follows:

[0057] Using the video capture device to shoot a smoke and fire detection video;

[0058] The client obtains the video stream address of the fire and smoke detection video, obtains the fire and smoke detection video according to the video stream address by using a real-time streaming protocol, and reads at least one frame of the fire and smoke detection video by using the VideoCapture class of OpenCV to obtain a picture to be detected;

[0059] The client accesses the smoke and fire detection training model through the gRPC service and parses the smoke and fire detection training model to obtain the model name and image format requirements of the smoke and fire detection training model;

[0060] The client preprocesses the image to be detected according to the image format requirement to obtain the image to be inferred;

[0061] The client creates model input data of a preset type according to the image to be inferred;

[0062] The client predefines a model output target set;

[0063] The client accesses the graphics processor through the gRPC service, and sends the model input data, the model output target set and the result callback function to the smoke and fire detection training model;

[0064] The smoke and fire detection training model detects the model input data based on the model output target set to obtain an image inference result, and sends the image inference result to the client according to the result callback function;

[0065] The client filters the image reasoning results according to a preset threshold, and packages the filtered image reasoning results into serialized fireworks data using the rapidjson tool and reports them to the alarm platform.

[0066] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0067] (1) The fire and smoke detection system based on C / S architecture in the example of the embodiment of this specification adopts cloud deployment to deploy the fire and smoke detection training model on a remote high-performance GPU server. The client is deployed with a video capture device and intelligent inspection equipment. The client and the server adopt grpc communication method, which can support the inference output of batch images.

[0068] (2) The fire and smoke detection system based on C / S architecture in the example of the embodiment of this specification has a client functional module including six interfaces: image reading, image preprocessing, model input, model output, model reasoning and post-processing. The client functional module is encapsulated into three C interfaces through the .so dynamic library for calling by the go language to realize the reporting of alarm data and alarm images.

[0069] (3) The fireworks detection method exemplified in the embodiments of this specification can detect fireworks continuously for a long time in high temperature, high pressure and corrosive environment, and has strong environmental adaptability; the captured fireworks detection images are pushed to the cloud inference server through cloud networking, so that the mobile detection area is not restricted by the local area network, and the detection device is more lightweight; the use of grpc service for communication is more suitable for batch processing of fireworks detection image data or video streams; the fireworks detection process is clear and the subsequent maintenance cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0071] Figure 1 A schematic diagram of the structure of a smoke and fire detection system based on a C / S architecture provided in an embodiment of this specification;

[0072] Figure 2 A flowchart of a smoke and fire detection method provided in an embodiment of this specification. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this application will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] First, the technical concept of the technical solution disclosed by the present invention is explained. At present, with the development of highly automated production in industrial parks, many equipment in industrial parks are operated under high temperature, high pressure and corrosive environments. This makes the equipment in industrial parks prone to dangerous accidents such as fire and explosion. Therefore, it is necessary to accurately, timely and effectively detect fireworks in the early stage of an accident, give early warning, and minimize the disasters of dangerous accidents such as fire and explosion.

[0075] Most existing fire monitoring equipment detects fireworks through fireworks sensors and surveillance videos. There are generally problems such as high false alarm rates, blind spots in monitoring, and poor environmental adaptability. It cannot meet the requirements of accurate, timely and effective fireworks detection in high temperature, high pressure and corrosive environments 24 hours a day without blind spots.

[0076] Therefore, it is now necessary to use advanced image processing and analysis technology, take the various problems encountered in fireworks warning practice as the starting point, combine innovative fireworks video recognition processes and processing methods, and develop fireworks detection and recognition technology based on video image intelligent analysis, strengthen fireworks monitoring and alarm, and prevent the occurrence of large-scale fire accidents.

[0077] The present invention provides a novel smoke and fire detection system based on a C / S architecture, and a smoke and fire detection method, device and medium based on the smoke and fire detection system. The technical solutions provided by each embodiment of the present application are described in detail below in conjunction with the accompanying drawings of the specification.

[0078] Figure 1 A schematic diagram of a smoke and fire detection system based on a C / S architecture provided in an embodiment of this specification. Figure 1 As shown, a fire and smoke detection system based on a C / S architecture is shown, wherein the system includes a cloud server and a client, the cloud server includes at least a graphics processor and a gRPC port, wherein the graphics processor is provided with at least a fire and smoke detection training model, and the fire and smoke detection training model is provided in the graphics processor using the Nvidia triton architecture; the client includes at least a video capture device and an intelligent inspection device, wherein the video capture device is provided in the intelligent inspection device, the video capture device is at least used to shoot fire and smoke detection videos, and the intelligent inspection device is at least used to load the video capture device and drive the video capture device to perform inspections; the cloud processor communicates with the client through the gRPC port, and the client accesses the fire and smoke detection training model through the gRPC service.

[0079] In this embodiment, the video capture device uses a camera, and the intelligent inspection device uses a robot. Of course, it is understandable that the video capture device can also use other devices such as cameras, as long as it can shoot fireworks detection videos, and the intelligent inspection device can also use other smart machines with drivers, as long as it can load the video capture device and drive the video capture device to inspect a certain area according to a preset route or under real-time control.

[0080] The fireworks detection system of the example in this embodiment adopts a cloud-based networked C / S architecture, and deploys the trained fireworks detection training model TensorFlow on a high-performance server in a cloud-based deployment manner. The robot platform (client) interacts with the cloud server through gRPC, and decodes the video data captured by the robot camera into frames of images and pushes them to the inference server in the cloud server. The cloud server feeds back the inference results to the client, and the client post-processes the inference results, and reports the data such as the number of fireworks ignition points, fireworks coordinates, fireworks types, and fireworks level assessment scores to the alarm platform to realize intelligent detection of fireworks.

[0081] In this embodiment, the client also includes six parts: an image reading module, an image preprocessing module, a model input module, a model output module, a model reasoning module and a reasoning result post-processing module.

[0082] The image reading module is used to obtain the video stream address of the fire and smoke detection video, obtain the fire and smoke detection video according to the video stream address using the real-time streaming protocol RTSP (Real Time Streaming Protocol), and use the VideoCapture class of OpenCV to read at least one frame of the fire and smoke detection video to obtain the image to be detected. In this embodiment, the image to be detected is in mat format.

[0083] The image preprocessing module is used to obtain the image to be detected in mat format, access the fireworks detection training model through gRPC, parse the fireworks detection training model, obtain the model name and image format requirements of the fireworks detection training model, and preprocess the image to be detected according to the image format requirements to obtain the image to be inferred.

[0084] In this embodiment, the picture format requirement may include at least one of the following: the length of the picture, the width of the picture, the color of the picture, and the number of channels of the picture.

[0085] In this embodiment, preprocessing the image to be detected may at least include the following steps: first, using the cvColor function of OpenCV to perform color conversion on the image to be detected; then, using the resize function to scale the image to be detected to a size that meets the image format requirements according to the size of the input image required by the model, and finally copying the image to be inferred obtained after preprocessing into a pre-specified cache variable through memcpy for ready call. In this embodiment, the image to be detected is scaled to a 3-channel 320*320 size.

[0086] The model input module is used to create a preset type of model input data based on the image to be inferred, that is, according to the model name of the parsed fireworks detection training model and the image format requirements such as length, width, color, number of channels, etc., to create Nvidia trition-client type model input data as the input of the inference interface in the cloud processor.

[0087] The model output module is used to predefine the model output target set, that is, to create a model output target set of the Nvidia trition-client type as the output of the inference interface in the cloud processor.

[0088] In this embodiment, the model output target set may include at least one of the following target items: the number of fireworks ignition points, fireworks coordinates, fireworks types, and fireworks level assessment scores.

[0089] The model inference module is used to access the graphics processor through the gRPC service and send the model input data, model output target set and result callback function to the fireworks detection training model.

[0090] The inference result post-processing module is used to obtain the image inference results and filter the image inference results according to the preset threshold, filtering out the results with fireworks level assessment scores lower than the preset threshold. The filtered image inference results are packaged into serialized fireworks data using the rapidjson tool and reported to the alarm platform.

[0091] In this embodiment, the image reasoning result may include at least one of the following: the number of fireworks ignition points, fireworks coordinates, fireworks type (open flame or smoke), and fireworks level assessment score.

[0092] Figure 2 A flow chart of a method for detecting fireworks provided in an embodiment of this specification. Figure 2 As shown, the smoke and fire detection method includes:

[0093] Using a video capture device to shoot fireworks detection video;

[0094] The client obtains the video stream address of the fireworks detection video, obtains the fireworks detection video using the real-time streaming protocol according to the video stream address, and uses the VideoCapture class of OpenCV to read at least one frame of the fireworks detection video to obtain the image to be detected;

[0095] The client accesses the fireworks detection training model through the gRPC service and parses the fireworks detection training model to obtain the model name and image format requirements of the fireworks detection training model;

[0096] The client preprocesses the image to be detected according to the image format requirements to obtain the image to be inferred;

[0097] The client creates model input data of a preset type based on the image to be inferred;

[0098] The client predefines the model output target set;

[0099] The client accesses the GPU through the gRPC service and sends the model input data, model output target set, and result callback function to the fireworks detection training model;

[0100] The fireworks detection training model detects the model input data based on the model output target set, obtains the image inference result, and sends the image inference result to the client according to the result callback function;

[0101] The client filters the image inference results according to the preset threshold, and uses the rapidjson tool to package the filtered image inference results into serialized fireworks data and report them to the alarm platform.

[0102] In this embodiment, the image to be detected is in mat format.

[0103] In this embodiment, the image format requirement includes at least one of the following: the length of the image, the width of the image, the color of the image, and the number of channels of the image.

[0104] In this embodiment, preprocessing the image to be detected according to the image format requirements includes at least the following steps:

[0105] Use OpenCV's cvColor function to convert the color of the image to be detected;

[0106] Use the resize function to scale the image to be detected to a size that meets the image format requirements;

[0107] The preprocessed image to be inferred is copied into the pre-specified cache variable through memcpy.

[0108] In this embodiment, the model output target set includes at least one of the following target items: the number of fireworks ignition points, fireworks coordinates, fireworks types, and fireworks level assessment scores.

[0109] In this embodiment, the smoke and fire detection training model detects the model input data based on the model output target set to obtain the image inference result, which includes at least the following steps:

[0110] The fire detection training model detects the model input data item by item according to the target items in the model output target set.

[0111] If the model input data contains data that meets the target item, the target item is marked according to the preset value in the image reasoning result;

[0112] If the model input data does not contain data that meets the target item, the target item will be marked as empty in the image inference result.

[0113] For example, the target items included in the model output target set are: the number of fireworks ignition points, fireworks coordinates, fireworks types, and fireworks level assessment scores. Correspondingly, the image reasoning results also include: the number of fireworks ignition points, fireworks coordinates, fireworks types (open flames or smoke), fireworks level assessment scores, etc.

[0114] The fireworks detection training model detects the model input data. If there is no fireworks in the model input data, the number of fireworks ignition points, fireworks coordinates, fireworks types, and fireworks level assessment scores in the image inference results are empty. If there are fireworks in the model input data, the number of fireworks ignition points in the image inference results is marked as the specific number of ignition points, the fireworks coordinates are marked as the specific coordinates of each ignition point, the fireworks type is marked as open flame or smoke, and the fireworks level assessment score is marked as the specific score of the fireworks level.

[0115] Then, the image inference results are filtered according to the preset threshold, and the results with fireworks level assessment scores lower than the preset threshold are filtered out. The filtered image inference results are packaged into serialized fireworks data using the rapidjson tool and reported to the alarm platform.

[0116] In this embodiment, the smoke and fire detection method further includes:

[0117] In the Linux environment, the client is compiled into a .so dynamic link library, and the six model processing interfaces of the client are encapsulated into three C interfaces of client initialization, model reasoning and resource release for calling by the Go language to realize alarm reporting.

[0118] To facilitate understanding of the present invention, the specific contents of the above-mentioned smoke and fire detection system based on C / S architecture are further described below:

[0119] 1. Deploy the fire detection training model of the triton server on the 10.180.150.60 device, and provide port 8401 for communication between grpc-server and grpc-client.

[0120] 2. Use the Go language to call the C interface or C++ to directly call the initialization interface to pass in the robot camera rtsp video stream address, triton server address (this example uses 10.180.150.60:8401), and the fire detection model name fire_smoke; parse the model's meta data and configuration information.

[0121] 3. Use the Go language to call the model inference interface to preprocess the image, create the model input and output, request the tritonserver through grpc to obtain the inference results, and finally post-process the data to obtain the json serialized inference results.

[0122] 4. Call the resource release interface to complete the program resource release.

[0123] This embodiment also provides a smoke and fire detection device, wherein the device includes:

[0124] at least one processor; and,

[0125] a memory communicatively connected to the at least one processor; wherein,

[0126] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0127] Using the video capture device to shoot a smoke and fire detection video;

[0128] The client obtains the video stream address of the fire and smoke detection video, obtains the fire and smoke detection video according to the video stream address by using a real-time streaming protocol, and reads at least one frame of the fire and smoke detection video by using the VideoCapture class of OpenCV to obtain a picture to be detected;

[0129] The client accesses the smoke and fire detection training model through the gRPC service and parses the smoke and fire detection training model to obtain the model name and image format requirements of the smoke and fire detection training model;

[0130] The client preprocesses the image to be detected according to the image format requirement to obtain the image to be inferred;

[0131] The client creates model input data of a preset type according to the image to be inferred;

[0132] The client predefines a model output target set;

[0133] The client accesses the graphics processor through the gRPC service, and sends the model input data, the model output target set and the result callback function to the smoke and fire detection training model;

[0134] The smoke and fire detection training model detects the model input data based on the model output target set to obtain an image inference result, and sends the image inference result to the client according to the result callback function;

[0135] The client filters the image reasoning results according to a preset threshold, and packages the filtered image reasoning results into serialized fireworks data using the rapidjson tool and reports them to the alarm platform.

[0136] This embodiment further provides a non-volatile computer storage medium for smoke and fire detection based on a C / S architecture, storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0137] Using the video capture device to shoot a smoke and fire detection video;

[0138] The client obtains the video stream address of the fire and smoke detection video, obtains the fire and smoke detection video according to the video stream address by using a real-time streaming protocol, and reads at least one frame of the fire and smoke detection video by using the VideoCapture class of OpenCV to obtain a picture to be detected;

[0139] The client accesses the smoke and fire detection training model through the gRPC service and parses the smoke and fire detection training model to obtain the model name and image format requirements of the smoke and fire detection training model;

[0140] The client preprocesses the image to be detected according to the image format requirement to obtain the image to be inferred;

[0141] The client creates model input data of a preset type according to the image to be inferred;

[0142] The client predefines a model output target set;

[0143] The client accesses the graphics processor through the gRPC service, and sends the model input data, the model output target set and the result callback function to the smoke and fire detection training model;

[0144] The smoke and fire detection training model detects the model input data based on the model output target set to obtain an image inference result, and sends the image inference result to the client according to the result callback function;

[0145] The client filters the image reasoning results according to a preset threshold, and packages the filtered image reasoning results into serialized fireworks data using the rapidjson tool and reports them to the alarm platform.

[0146] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0147] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0148] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] The present invention is a method, device (system), and computer program product according to an embodiment of the present invention.

[0150] The flowcharts and / or block diagrams of the products are described. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0153] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0154] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0155] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0156] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0157] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A fire and smoke detection system based on C / S architecture, characterized in that: The system comprises: A cloud server, the cloud server comprising at least a graphics processor and a gRPC port, wherein the graphics processor is provided with at least a smoke and fire detection training model, and the smoke and fire detection training model is provided in the graphics processor using an Nvidia triton architecture; A client, the client at least comprising a video capture device and an intelligent inspection device, wherein the video capture device is arranged in the intelligent inspection device, the video capture device is at least used to shoot a smoke and fire detection video, and the intelligent inspection device is at least used to load the video capture device and drive the video capture device to perform inspection; The cloud server communicates with the client through the gRPC port, and the client accesses the smoke and fire detection training model through the gRPC service; The client also includes: An image reading module, the image reading module is used to obtain the video stream address of the fireworks detection video, obtain the fireworks detection video according to the video stream address using a real-time streaming protocol, and use the VideoCapture class of OpenCV to read at least one frame of the fireworks detection video to obtain a picture to be detected; An image preprocessing module, the image preprocessing module is used to parse the smoke and fire detection training model to obtain the model name and image format requirements of the smoke and fire detection training model, and to preprocess the image to be detected according to the image format requirements to obtain the image to be inferred; A model input module, the model input module is used to create model input data of a preset type according to the picture to be inferred; A model output module, wherein the model output module is used to predefine a model output target set; A model reasoning module, the model reasoning module is used to access the graphics processor through the gRPC service, and send the model input data, the model output target set and the result callback function to the smoke and fire detection training model; The inference result post-processing module is used to filter the image inference results according to a preset threshold, and package the filtered image inference results into serialized fireworks data using the rapidjson tool and report them to the alarm platform.

2. A method for detecting fireworks based on the fireworks detection system based on C / S architecture according to claim 1, characterized in that: The method comprises: Using the video capture device to shoot a smoke and fire detection video; The client obtains the video stream address of the fire and smoke detection video, obtains the fire and smoke detection video according to the video stream address by using a real-time streaming protocol, and reads at least one frame of the fire and smoke detection video by using the VideoCapture class of OpenCV to obtain a picture to be detected; The client accesses the smoke and fire detection training model through the gRPC service and parses the smoke and fire detection training model to obtain the model name and image format requirements of the smoke and fire detection training model; The client preprocesses the image to be detected according to the image format requirement to obtain the image to be inferred; The client creates model input data of a preset type according to the image to be inferred; The client predefines a model output target set; The client accesses the graphics processor through the gRPC service, and sends the model input data, the model output target set and the result callback function to the smoke and fire detection training model; The smoke and fire detection training model detects the model input data based on the model output target set to obtain an image inference result, and sends the image inference result to the client according to the result callback function; The client filters the image reasoning results according to a preset threshold, and packages the filtered image reasoning results into serialized fireworks data using the rapidjson tool and reports them to the alarm platform.

3. The method for detecting smoke and fire according to claim 2, wherein: The image format requirement includes at least one of the following: the length of the image, the width of the image, the color of the image, and the number of channels of the image.

4. The method for detecting smoke and fire according to claim 2, wherein: The preprocessing of the to-be-detected picture according to the picture format requirement at least comprises the following steps: Use the cvColor function of OpenCV to perform color conversion on the image to be detected; Use the resize function to scale the image to be detected to a size that meets the requirements of the image format; The preprocessed image to be inferred is copied into the pre-specified cache variable through memcpy.

5. The method for detecting smoke and fire according to claim 2, wherein: The model output target set includes at least one of the following target items: the number of fireworks ignition points, fireworks coordinates, fireworks types, and fireworks level assessment scores.

6. The method for detecting smoke and fire according to claim 5, characterized in that: The smoke and fire detection training model detects the model input data based on the model output target set to obtain an image inference result, and at least includes the following steps: The fire and smoke detection training model detects the model input data item by item according to the target items in the model output target set, If the model input data contains data that meets the target item, the target item is marked according to a preset value in the image reasoning result; If the model input data does not contain data that meets the target item, the target item is marked as empty in the image reasoning result.

7. The method for detecting smoke and fire according to claim 2, wherein: The method further comprises: In the Linux environment, the client is compiled into a .so dynamic link library, and the 6 model processing interfaces of the client are encapsulated into 3 C interfaces of client initialization, model reasoning and resource release for calling by the Go language to realize alarm reporting; among them, the 6 model processing interfaces include: image reading, image preprocessing, model input, model output, model reasoning and post-processing 6 interfaces.

8. A smoke and fire detection device based on the smoke and fire detection system based on C / S architecture according to claim 1, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Using the video capture device to shoot a smoke and fire detection video; The client obtains the video stream address of the fire and smoke detection video, obtains the fire and smoke detection video according to the video stream address by using a real-time streaming protocol, and reads at least one frame of the fire and smoke detection video by using the VideoCapture class of OpenCV to obtain a picture to be detected; The client accesses the smoke and fire detection training model through the gRPC service and parses the smoke and fire detection training model to obtain the model name and image format requirements of the smoke and fire detection training model; The client preprocesses the image to be detected according to the image format requirement to obtain the image to be inferred; The client creates model input data of a preset type according to the image to be inferred; The client predefines a model output target set; The client accesses the graphics processor through the gRPC service, and sends the model input data, the model output target set and the result callback function to the smoke and fire detection training model; The smoke and fire detection training model detects the model input data based on the model output target set to obtain an image inference result, and sends the image inference result to the client according to the result callback function; The client filters the image reasoning results according to a preset threshold, and packages the filtered image reasoning results into serialized fireworks data using the rapidjson tool and reports them to the alarm platform.

9. A non-volatile computer storage medium for smoke and fire detection based on the smoke and fire detection system based on C / S architecture according to claim 1, storing computer executable instructions, characterized in that: The computer executable instructions are configured to: Using the video capture device to shoot a smoke and fire detection video; The client obtains the video stream address of the fire and smoke detection video, obtains the fire and smoke detection video according to the video stream address by using a real-time streaming protocol, and reads at least one frame of the fire and smoke detection video by using the VideoCapture class of OpenCV to obtain a picture to be detected; The client accesses the smoke and fire detection training model through the gRPC service and parses the smoke and fire detection training model to obtain the model name and image format requirements of the smoke and fire detection training model; The client preprocesses the image to be detected according to the image format requirement to obtain the image to be inferred; The client creates model input data of a preset type according to the image to be inferred; The client predefines a model output target set; The client accesses the graphics processor through the gRPC service, and sends the model input data, the model output target set and the result callback function to the smoke and fire detection training model; The smoke and fire detection training model detects the model input data based on the model output target set to obtain an image inference result, and sends the image inference result to the client according to the result callback function; The client filters the image reasoning results according to a preset threshold, and packages the filtered image reasoning results into serialized fireworks data using the rapidjson tool and reports them to the alarm platform.

Images