Production line control method and device based on data perception, equipment and medium

Abstract

The invention discloses a production line control method and device based on data perception, equipment and a medium. The method comprises the following steps: acquiring current smoke concentration data of a target area of a production line; determining a target object recognition result of the target area based on the video stream data of the target area under the condition that the current smoke concentration data meets the accident judgment condition; the target object recognition result is at least used for representing whether a living object exists in the target area; based on the target object identification result of the target area, determining an accident execution strategy for the target area, and executing the accident execution strategy of the target area; the accident execution strategy at least comprises a strategy used for controlling the operation state of the target area.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a production line control method, device, equipment and medium based on data perception. Background Art

[0002] For people working in flexible production lines, their safety needs to be ensured. In the prior art, although there are some safety monitoring systems on the production equipment of the flexible production line to provide protection for the various production equipment of the flexible production line, their functions have not been fully integrated, resulting in imperfect functions of the existing flexible production line safety monitoring system. For example, when an abnormal failure occurs in a piece of production equipment in the flexible production line, which leads to fire or smoke spreading, it is often necessary to evacuate the people in the flexible production line and cut off the power of all relevant power supply systems of the flexible production line as soon as possible, otherwise the electrical fire may spread. If the power supply is cut off before someone leaves the flexible production line, it may increase the difficulty of their escape. For example, the power is cut off and the lighting system is turned off, resulting in extremely poor light. The obstruction of the complex equipment of the flexible production line may make it difficult for the staff to find the escape exit or escape direction, or the electric fire door may be closed, resulting in someone being unable to escape, causing chaos in escape management. Summary of the invention

[0003] In view of this, the embodiments of the present application at least provide a production line control method, device, equipment and medium based on data perception.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a production line control method based on data perception, the method comprising:

[0006] Obtain current smoke concentration data in the target area of ​​the production line;

[0007] In the case where the current smoke concentration data meets the accident judgment condition, determining a target object recognition result of the target area based on the video stream data of the target area; the target object recognition result is at least used to characterize whether there is a living object in the target area;

[0008] Based on the target object recognition result of the target area, an accident execution strategy for the target area is determined, and the accident execution strategy for the target area is executed; the accident execution strategy at least includes a strategy for controlling the operating state of the target area.

[0009] In a second aspect, an embodiment of the present application provides a production line control device based on data perception, the production line control device comprising:

[0010] An acquisition module is used to obtain the current smoke concentration data of the target area of ​​the production line;

[0011] an identification module, configured to determine a target object identification result of the target area based on the video stream data of the target area when the current smoke concentration data meets the accident judgment condition; the target object identification result is at least used to characterize whether there is a living object in the target area;

[0012] An execution module is used to determine an accident execution strategy for the target area based on the target object recognition result of the target area, and execute the accident execution strategy for the target area; the accident execution strategy at least includes a strategy for controlling the operating state of the target area.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which implement some or all of the steps in the above method when executed by a processor.

[0016] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0018] Figure 1 A schematic diagram of an implementation process of a production line control method based on data perception provided in an embodiment of the present application Figure 1 ;

[0019] Figure 2 A schematic diagram of an implementation process of a production line control method based on data perception provided in an embodiment of the present application Figure 2 ;

[0020] Figure 3 A schematic diagram of an implementation process of a production line control method based on data perception provided in an embodiment of the present application Figure 3 ;

[0021] Figure 4 A schematic diagram of an implementation process of a production line control method based on data perception provided in an embodiment of the present application Figure 4 ;

[0022] Figure 5 A schematic diagram of an implementation process of a production line control method based on data perception provided in an embodiment of the present application Figure 5 ;

[0023] Figure 6 A schematic diagram of the composition structure of a production line control device based on data perception provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a hardware entity of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0026] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0029] For people working in flexible production lines, it is necessary to ensure their safety in the flexible production lines. In the prior art, although there are some safety monitoring systems on the production equipment of the flexible production lines to provide protection for the various production equipment of the flexible production lines, their functions have not been fully integrated, resulting in imperfect functions of the existing flexible production line safety monitoring systems. For example, when a certain production equipment in the flexible production line has an abnormal failure, which leads to fire or smoke spreading, it is often necessary to evacuate the people in the flexible production line and cut off the power of all relevant power supply systems of the flexible production line as soon as possible, otherwise it may cause the spread of electrical fires. If the power supply is cut off before someone leaves the flexible production line, it may increase the difficulty of their escape. For example, the power is cut off and the lighting system is turned off, resulting in extremely poor light. The obstruction of the complex equipment of the flexible production line will make it difficult for the staff to find the escape exit or escape direction, or the electric fire door is closed, causing someone to be unable to escape, resulting in chaotic escape management. In addition, since the production equipment of the flexible production line is often not fixed, the location and type will change with the project plan, that is, the escape path may change, and the production environment is relatively complex, which further increases the difficulty of escape for the relevant personnel. The prior art urgently needs a solution to this technical problem.

[0030] In order to solve the above technical problems, the present application provides a production line control method based on data perception, which can be applied to the host computer of the production line. Figure 1 As shown, the above processing method can be implemented through steps S101 to S103:

[0031] Step S101, obtaining current smoke concentration data of a target area of ​​a production line.

[0032] Here, the production line refers to a flexible production line. The equipment in the flexible production line may change due to changes in the project, and the change in equipment refers to a change in the type of equipment and / or a change in the position of the equipment on the production line. The target area of ​​the production line may be any area on the production line. In some embodiments, the target area may also refer to an area on the production line where abnormalities are prone to occur.

[0033] In the embodiment of the present application, a smoke alarm can be pre-installed in the target area of ​​the production line, and the smoke alarm can collect the current smoke concentration data of the target area in real time. The current smoke concentration data refers to the concentration of smoke in the air, which can be expressed by the smoke ppm value.

[0034] Step S102, when the current smoke density data meets the accident judgment condition, based on the video stream data of the target area, determining the target object recognition result of the target area; the target object recognition result is at least used to characterize whether there is a living object in the target area. The living object may be a human object.

[0035] Here, the accident judgment condition refers to that the current smoke concentration data is greater than the preset concentration threshold and lasts for a certain period of time.

[0036] In the embodiment of the present application, when the current smoke concentration data meets the accident judgment condition, the host computer can determine that there is a safety accident in the target area of ​​the production line. At this time, it is necessary to obtain the video stream data of the target area, and determine whether there is a living object in the target area through the video stream data of the target area, that is, determine the target object recognition result of the target area. Among them, an image acquisition device is set on the target area, and the image acquisition device can acquire real-time video stream data of the target area, and send the real-time video stream data to the host computer after receiving the instruction of the host computer.

[0037] In some embodiments, the host computer can detect the video stream data of the target area through the trained liveness detection model to obtain the target object recognition result of the target area. The trained liveness detection model can be a deep neural network model.

[0038] Step S103, based on the target object recognition result of the target area, determine the accident execution strategy for the target area, and execute the accident execution strategy for the target area; the accident execution strategy at least includes a strategy for controlling the operating state of the target area.

[0039] In the embodiment of the present application, different accident execution strategies can be determined for different target object identification results. It can be understood that in the event of an accident in the target area, whether there are living objects in the target area determines the generation of the accident execution strategy. Exemplarily, when there are no living objects in the target area, the accident execution strategy is more inclined to protect the equipment in the production line from being damaged on the basis of controlling the accident; when there are living objects in the target area, the accident execution strategy prioritizes protecting the living objects from evacuating the target area as soon as possible.

[0040] In an embodiment of the present application, the accident execution strategy at least includes a strategy for controlling the operating state of the target area. Among them, the operating state of the target area can be the operating state of the equipment in the target area, and the equipment operating state includes a disconnected state and a normal operating state. It can be understood that whether there is a living object in the target area also determines the operating state of the target area. Exemplarily, when there is no living object in the target area, in order to protect the equipment in the production line from being damaged, all equipment in the target area can be controlled to be in a disconnected state; when there is a living object in the target area, in order to enable the living object to escape from the target area smoothly, the equipment used for lighting cannot be disconnected. Therefore, the disconnected state of the target area may also include a fully disconnected state and a partially disconnected state, and the partially disconnected state refers to disconnecting equipment that is not used for lighting.

[0041] In an embodiment of the present application, when the current smoke concentration data of the target area of ​​the production line meets the accident judgment condition, the target object recognition result of the target area is determined based on the video stream data of the target area; then, based on the target object recognition result of the target area, an accident execution strategy including at least one for controlling the operating state of the target area is determined, and the accident execution strategy of the target area is executed. In this way, when it is determined that an accident has occurred in the target area, the operating state of the target area can be controlled by whether there is a living object in the target area, so that when there is a living object in the target area, the difficulty of the living object to escape can be reduced, and when there is no living object in the target area, the damage caused by the accident to the equipment in the production line can be reduced.

[0042] In some embodiments, the video stream data includes at least two frames of image data; the “determining the target object recognition result of the target area based on the video stream data of the target area” in the above step S102 is implemented by steps S201 and S202:

[0043] Step S201 : performing living object recognition processing on the at least two frames of image data to obtain an image recognition result for each of the image data; the image recognition result is used to indicate whether a living object exists in the image data.

[0044] In the embodiment of the present application, a trained liveness recognition model can be used to perform liveness object recognition processing on at least two frames of image data to obtain an image recognition result for each image data. Exemplarily, the trained liveness recognition model can be a YOLO (You Only Look Once) model.

[0045] In some embodiments, the liveness recognition model can be trained to recognize human objects by the following steps:

[0046] S1. Collect target human body pictures to construct a dataset, and divide the constructed dataset into a training set, a test set, and a validation set.

[0047] S2. Use the k-means clustering algorithm to cluster the images in the dataset and obtain K prior boxes.

[0048] S3. Add 1×1 convolution kernels before the first and second 3×3 convolution layers of the Backbone of the YOLOv5 algorithm; add Non-local attention mechanism modules before the third and fourth BottleneckCSP of the Backbone of the YOLOv5 algorithm.

[0049] S4. Use the CSPNet structure consisting of Bottleneck and standard convolutional layers in the Backbone of the YOLOv5 algorithm and add an Output detection layer; reference the PANet structure in the Neck of the YOLOv5 algorithm; generate an improved YOLOv5 algorithm.

[0050] S5, train and test the improved YOLOv5 algorithm on the established target human dataset. Further, in step S1, collect the target human image, convert the obtained image format into .jpg format, use labelImg software to annotate the image type, output the format into .xml file, and make it into a standard VOC dataset format, then randomly divide the constructed dataset into training set, test set and validation set in a ratio of 8:1:1, and annotate the target human name to complete the construction of the target human dataset.

[0051] Furthermore, in step S5, the training and testing on the target human body data set specifically includes:

[0052] (1) Set the parameters of the improved YOLOv5 algorithm: the training step size is set to 300, the initial learning rate is set to 0.01, and the batch_size is set to 1.

[0053] (2) Use the k-means clustering algorithm to obtain K target anchor values ​​and modify the original anchorbox to the new anchorbox after clustering.

[0054] (3) The input image is first subjected to Mosaic data enhancement and image normalization processing, and the image used for training is normalized to a size of 640×640.

[0055] (4) The Backbone part slices and convolves the image to obtain a feature map.

[0056] (5) Feature information is extracted from the feature map through the Neck part. In the Neck part, the image is first upsampled to extract image features, and then downsampled to fuse feature information of different scales. Finally, the detection result is output from the output end to obtain the detection model.

[0057] (6) Use the validation set to verify the performance of the model.

[0058] Step S201: determining a target object recognition result of the target area based on an image recognition result of each of the image data.

[0059] In some embodiments, Figure 2 As shown, the above step S201 can be implemented through steps S2011 to S2013:

[0060] Step S2011: determining a first quantity of target image data in which no living object exists based on the image recognition result of each of the image data.

[0061] In the embodiment of the present application, a first quantity of target image data in at least two frames of image data may be determined, wherein the target image data refers to image data in which the image recognition result indicates that there is no living object.

[0062] Step S2012: When the first number is greater than or equal to a first threshold and the first number of target image data is continuous in image acquisition time, determine that there is no living object in the target area.

[0063] Step S2013: When the first number is less than a first threshold, determine that there is a living object in the target area.

[0064] In the embodiment of the present application, the first number can be determined by a counter. Exemplarily, for each image data, if it is determined that the image recognition result of the current image data indicates that there is no living object, then the counter is incremented by 1, and if it is determined that the image recognition result of the current image data indicates that there is a living object, then the counter is cleared to 0, and the above steps are repeated until all image data are recognized.

[0065] In some embodiments, the condition for determining that there is no living object in the target area may also be that the first number is greater than or equal to the first threshold, the first number of target image data is continuous in image acquisition time, and the image acquisition time of the last target image data is the last frame of image data in at least two frames of image data.

[0066] In the embodiment of the present application, in order to improve the accuracy of determining whether there is a living object in the target area as much as possible, the value of the first threshold can be set to be larger. For example, the first threshold can be 60. That is, if the existence of a living object cannot be identified in the 60 frames of image data after the video stream data of the target area, it is determined that there is no living object in the target area.

[0067] In some embodiments, the above step S103 of “determining the accident execution strategy for the target area based on the target object recognition result of the target area” can be implemented by one of step S1031 and step S1032:

[0068] Step S1031 : when the target object recognition result indicates that there is no living object in the target area, determine a first strategy for controlling the operation state of the target area to be in a fully disconnected state.

[0069] It is understandable that when the target object recognition result indicates that there is no living object in the target area, it can be determined that the purpose of the current accident execution strategy is to reduce the loss of the equipment caused by the accident. At this time, the first strategy of controlling the operating state of the target area to be in a completely disconnected state can be determined.

[0070] In some embodiments, after determining the first strategy, the host computer can directly control the operation state of the target area to be in a fully disconnected state based on the first strategy. In other embodiments, the host computer can send the first strategy to the controller of the production line, and then control the operation state of the target area to be in a fully disconnected state through the controller of the production line. The controller can be a programmable logic controller (PLC).

[0071] Step S1032, when the target object recognition result indicates that there is a living object in the target area, determine a second strategy of outputting an alarm signal, determining an escape path for the living object in the target area, and controlling the operating state of the target area to be in a partially disconnected state.

[0072] It is understandable that when the target object recognition result indicates that there is a living object in the target area, it can be determined that the purpose of the current accident execution strategy is to enable the living object to escape the target area smoothly. At this time, a second strategy can be determined. The second strategy includes outputting an alarm signal, determining an escape path for the living object in the target area, and controlling the operating state of the target area to be in a partially disconnected state. The partially disconnected state means that the equipment in the target area except for lighting is in a disconnected state.

[0073] In some embodiments, Figure 3 As shown, the "determining an escape path for the living object in the target area" in the above step S1032 can be implemented through steps S301 to S303:

[0074] Step S301, obtaining real-time map information of the production line; the real-time map information is determined based on the real-time attribute information of each production equipment of the production line and the coordinate range information of the production line.

[0075] It is understandable that because the production line is a flexible production line, the equipment in the flexible production line will change depending on the current project. Therefore, it is necessary to obtain real-time map information of the production line.

[0076] In the embodiment of the present application, the real-time map information of the production line is determined based on the real-time attribute information of each production equipment of the production line and the coordinate range information of the production line. The real-time attribute information of the production equipment includes the location information and size information of the equipment. Because the equipment in the production line will change, the coordinate range information of the production line also needs to be determined in real time based on the real-time attribute information of each production equipment.

[0077] Step S302: determining the position information of the living object in the real-time map information based on the image data containing the living object.

[0078] In the embodiment of the present application, the location information of the living object in the real-time map information can be determined based on the location information of the device in the image data of the living object.

[0079] In some embodiments, the location information of the living object in the real-time map information can be determined based on the coordinate information of the living object in the image data and the conversion relationship between the image coordinate system and the actual geographic coordinate system.

[0080] Step S303: determining at least one escape path of the living object based on the location information of the living object and the real-time map information of the production line.

[0081] In the embodiment of the present application, at least one escape path closest to the living object can be determined for the living object based on the location information of the living object and the real-time map information of the production line.

[0082] In some embodiments, after the above step S303, the escape path of the living object may also be updated; Figure 4 As shown, the solution of “updating the escape path of the living object” can be implemented through steps S401 to S403:

[0083] Step S401, when the number of the escape paths is at least two, based on the coordinate range information of the escape paths and the position information of the living objects, the number of living objects on each of the escape paths is determined. The living objects may be human objects.

[0084] In the embodiment of the present application, when there are two escape paths, the number of living objects on the escape paths can be adjusted to reduce the difficulty of the living objects escaping from the target area due to the large number of living objects on the escape paths. Therefore, the number of living objects on each escape path can be determined based on the location information of the living objects and the coordinate range information of the escape paths.

[0085] Exemplarily, if the position information of the living object 1 is within the coordinate range information of the escape route 1 , it is determined that the living object 1 is on the escape route 1 .

[0086] Step S402: for a target escape path in which the number of living objects is greater than or equal to a second threshold, determine an escape extension time for a second number of living objects on the target escape path when changing the escape path; the difference between the number of living objects on the target escape path and the second number is less than the second threshold.

[0087] In the embodiment of the present application, it is possible to first determine whether the number of living objects on each escape path is greater than a second threshold value. If so, it means that the number of living objects on the escape path is large, and the number of people on the escape path can be adjusted. That is, the escape extension time when the second number of living objects on the target escape path is changed is determined.

[0088] Here, the second number of living objects is randomly selected by the host computer from the living objects in the target escape path, and the number of living objects on the target escape path after selection needs to be less than a second threshold, that is, the difference between the number of living objects on the target escape path and the second number is less than the second threshold.

[0089] In the embodiment of the present application, after determining the second number of living objects on the target escape path, the extended escape time of the second number of living objects after changing the escape path can be predicted. It can be understood that the escape path is the optimal route for each living object to escape from the target area, that is, the escape time is minimized, so when the living object changes the escape route, the escape time will increase, so it is necessary to determine whether the extended escape time of the second number of living objects after changing the escape path is too long.

[0090] Step S403: when the escape extension time is less than a third threshold, changing the escape path of the living object of the second data; and the number of the living objects on the escape path after the change is less than the second threshold.

[0091] In the embodiment of the present application, when the escape extension time is less than the third threshold, it indicates that the escape extension time of the living object that changes the escape path is short, so the escape path can be changed for the living object.

[0092] It should be noted that the number of living objects on the escape path after the change is smaller than the second threshold value, so that the number of living objects on the escape path can be smaller, thereby reducing the difficulty of escaping for the living objects.

[0093] In some embodiments, Figure 5 As shown, "whether the current smoke concentration data meets the accident judgment condition" can be achieved through steps S501 and S502:

[0094] Step S501, when the current smoke density data is greater than or equal to a fourth threshold, determining a duration during which the current smoke density data is greater than or equal to the fourth threshold.

[0095] In the embodiment of the present application, it can be determined whether the current smoke density data is greater than or equal to the fourth threshold. If the current smoke density data is greater than or equal to the fourth threshold, it indicates that there is a suspected fire in the target area. At this time, it is necessary to determine the duration of the suspected fire in the target area, that is, to determine the duration of the current smoke density data being greater than or equal to the fourth threshold.

[0096] Step S502: when the duration is greater than or equal to a fifth threshold, determining that the current smoke density data meets an accident judgment condition.

[0097] In an embodiment of the present application, when it is determined that there is a suspected fire in the target area, it is necessary to determine whether the duration of the suspected fire in the target area is greater than or equal to the fifth threshold. When the duration is greater than or equal to the fifth threshold, it can be determined that the current smoke concentration data meets the accident judgment conditions.

[0098] The above-mentioned data-aware production line control method is described below in conjunction with a specific embodiment. For ease of understanding, a possible process applicable to the embodiment of the present application is introduced below using a fire accident as an example. However, it is worth noting that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.

[0099] The embodiment of the present application provides a production line control method based on data perception, including steps S200 to S230:

[0100] Step S200, in combination with the environment of the flexible production line, updating the environmental map of the flexible production line in real time, planning all available escape routes of the flexible production line in real time, using the smoke alarm set in the target area of ​​the flexible production line, collecting the current smoke concentration data of the target area at a preset time interval, judging whether the current smoke concentration data is greater than or equal to a first preset threshold, judging whether there is a suspected fire in the target area according to the judgment result, calculating the first duration of the suspected fire in the target area according to the preset time interval, and judging whether there is a fire in the target area according to the first duration.

[0101] It should be noted that before step S200, the following steps are included:

[0102] A first preset threshold (the fourth threshold in the above embodiment), a second preset threshold (the fifth threshold in the above embodiment), a third preset threshold (the first threshold in the above embodiment), a fourth preset threshold (the second threshold in the above embodiment), a preset time interval, a preset size of each production equipment of the flexible production line (size information of the production equipment in the above embodiment) and a preset coordinate range of the flexible production line (coordinate range information of the production line in the above embodiment) are preset.

[0103] It can be understood that the first preset threshold, the second preset threshold, the third preset threshold, the fourth preset threshold, the fifth preset threshold, the preset time interval, the preset dimensions of each production equipment of the flexible production line and the preset coordinate range of the flexible production line can be set according to actual needs. The present invention does not impose specific restrictions on the values ​​of the first preset threshold, the second preset threshold, the third preset threshold, the fourth preset threshold, the fifth preset threshold, the preset time interval, the preset dimensions of each production equipment of the flexible production line and the preset coordinate range of the flexible production line, as long as it is applicable to the production line control method based on data perception proposed in the present invention.

[0104] Preferably, the present invention sets the first preset threshold to 70 ppm, the present invention sets the second preset threshold to 30 seconds, the present invention sets the third preset threshold to 60, the present invention sets the preset time interval to 1 second, the present invention sets the fourth preset threshold to 3 people, and the present invention sets the fifth preset threshold to 10 seconds.

[0105] The above settings are obtained by the technicians of the present invention through a large number of experiments, and can better detect whether there is a suspected fire in the target area, or whether there is a fire, or whether there is a target human body, and perform data fusion analysis, and then determine whether to control all the electric control switches supplying power to the target area to be disconnected, further improving the intelligence and reliability of the present invention.

[0106] Specifically, the determining whether the current smoke density data is greater than or equal to a first preset threshold, and determining whether there is a suspected fire in the target area according to the determination result, includes:

[0107] If the current smoke density data is greater than or equal to the first preset threshold, it is determined that there is a suspected fire in the target area;

[0108] If the current smoke concentration data is less than the first preset threshold, it is determined that there is no suspected fire in the target area.

[0109] Specifically, recording a first duration of a suspected fire situation in the target area according to a preset time interval, and judging whether there is a fire situation in the target area according to the first duration, includes:

[0110] If the current smoke concentration data is greater than or equal to the first preset threshold, the first counter is controlled to add 1; if the current smoke concentration data is less than the first preset threshold, the first counter is controlled to be cleared; based on the accumulated value of the first counter and the preset time interval, the first duration of the suspected fire in the target area is calculated.

[0111] Specifically, recording a first duration of a suspected fire situation in the target area according to a preset time interval, and judging whether there is a fire situation in the target area according to the first duration, includes:

[0112] Determine whether the first duration (the duration in the above embodiment) is greater than or equal to a second preset threshold;

[0113] If so, it is determined that there is a fire in the target area;

[0114] If not, it is determined that there is no fire in the target area.

[0115] It is understandable that if the first duration is greater than or equal to 30 seconds, the present invention determines that the suspected fire situation lasts too long and determines that a fire situation exists.

[0116] If the first duration is less than 30 seconds, the present invention determines that the suspected fire situation lasts for a short time, and the current smoke concentration data has dropped below the first preset threshold, which may be caused by environmental fluctuations, and determines that there is no fire situation.

[0117] The step S200 further includes:

[0118] By utilizing the positioning devices, preset dimensions, and preset coordinate ranges of the production equipment on the flexible production line, an environmental map of the flexible production line is constructed and updated in real time, thereby obtaining all accurate escape paths for the flexible production line in real time.

[0119] It can be understood that the preset coordinate range of the flexible production line is the preset coordinate range of the entire flexible production line.

[0120] Step S210: If there is a suspected fire in the target area, a camera arranged above the target area is used to collect a current frame image of the target area of ​​the flexible production line at a preset time interval, and the current frame image is analyzed by the YOLOV5 algorithm to determine whether there is a target image defined as a target human body in the current frame image. If so, the optimal path is recommended based on all available escape paths of the flexible production line and the current position of the personnel, and the escape path is sent to the staff of the flexible production line through the cloud server.

[0121] It should be noted here that the YOLOV5 algorithm is used to analyze the current frame image to determine whether there is a target image defined as a target human body in the current frame image, including:

[0122] Step S1: collect target human body pictures to construct a data set, and divide the constructed data set into a training set, a test set, and a validation set.

[0123] Step S2: Cluster the images in the data set using the k-means clustering algorithm to obtain K prior boxes.

[0124] Step S3: Add 1×1 convolution kernels before the first and second 3×3 convolution layers of the Backbone of the YOLOv5 algorithm; add Non-local attention mechanism modules before the third and fourth BottleneckCSP of the Backbone of the YOLOv5 algorithm.

[0125] Step S4: Use the CSPNet structure consisting of Bottleneck and standard convolutional layers in the Backbone of the YOLOv5 algorithm and add an Output detection layer; reference the PANet structure in the Neck of the YOLOv5 algorithm; and generate an improved YOLOv5 algorithm.

[0126] Step S5, train and test the improved YOLOv5 algorithm on the established target human dataset. Further, in step S1, collect the target human image, convert the obtained image format into .jpg format, use labelImg software to annotate the image type, output the format into .xml file, and make it into a standard VOC dataset format, then randomly divide the constructed dataset into training set, test set and validation set in a ratio of 8:1:1, and annotate the target human name to complete the construction of the target human dataset.

[0127] Furthermore, in step S5, the training and testing on the target human body data set specifically includes:

[0128] (1) Set the parameters of the improved YOLOv5 algorithm: the training step size is set to 300, the initial learning rate is set to 0.01, and the batch_size is set to 1.

[0129] (2) Use the k-means clustering algorithm to obtain K target anchor values ​​and modify the original anchorbox to the new anchorbox after clustering.

[0130] (3) The input image is first subjected to Mosaic data enhancement and image normalization processing, and the image used for training is normalized to a size of 640×640.

[0131] (4) The Backbone part slices and convolves the image to obtain a feature map;

[0132] (5) Feature information is extracted from the feature map through the Neck part. In the Neck part, the image is first upsampled to extract image features, and then downsampled to fuse feature information of different scales. Finally, the detection result is output from the output end to obtain the detection model.

[0133] (6) Use the validation set to verify the performance of the model.

[0134] It should be noted here that the YOLOV5 algorithm is a prior art and the present invention will not go into details here.

[0135] Step S230: Determine whether all the electric switches supplying power to the target area are disconnected according to the determination result of step S210.

[0136] Specifically, the control of disconnecting or connecting all the electric switches supplying power to the target area can be operated through an automated control system, that is, through an integrated automated control system, such as a SCADA (Supervisory Control And Data Acquisition) system, instructions can be sent remotely to disconnect the electric switches in the target area, or to control all the electric switches supplying power to the target area to be connected. It is only necessary to ensure that the control system has accurately programmed all the electric switches in the target area; the control module can also be connected to the electric switch control, and the control module controls all the electric switches supplying power to the target area to be turned on or off.

[0137] Specifically, judging whether to control all electric control switches supplying power to the target area to be disconnected according to the judgment result of step S210 includes:

[0138] If a target image defined as a target human body exists in the current frame image, an alarm signal indicating that a target human body exists in the suspected fire area is output;

[0139] If there is no target image defined as a target human body in the current frame image, the current frame image is recorded as an image without a target human body, and the second counter is controlled to add 1; if there is a target image defined as a target human body in the current frame image, the second counter is controlled to be cleared, and it is determined whether the accumulated value of the second counter is greater than or equal to a third preset threshold value, and according to the determination result of step S210, it is determined whether all the electric control switches supplying power to the target area are controlled to be disconnected.

[0140] Specifically, the step of determining whether the accumulated value of the second counter (the first number in the above embodiment) is greater than or equal to a third preset threshold, and determining whether to control all the electric control switches supplying power to the target area to be disconnected according to the determination result of step S210, includes:

[0141] If the accumulated value of the second counter is greater than or equal to the third preset threshold, all the electric control switches supplying power to the target area are controlled to be disconnected.

[0142] Specifically, the step of determining whether the accumulated value of the second counter is greater than or equal to a third preset threshold value and determining whether to control all the electric control switches supplying power to the target area to be disconnected according to the determination result of step S210 further includes:

[0143] If the accumulated value of the second counter is less than the third preset threshold, the electric control switch supplying power to the target area is not disconnected, and a voice playback device arranged in the target area is used to output a voice alarm signal indicating that there is a suspected fire in the target area and the user should leave the target area immediately.

[0144] It can be understood that if the accumulated value of the second counter is less than the third preset threshold value, it proves that the target person still appears in the target area within a time of less than 1 minute, thereby preventing the power of the target area from being cut off when escaping late due to the target person being in a sleeping state or other reasons, thereby increasing the difficulty of escape, further improving the intelligence and usability of the present invention.

[0145] Specifically, the step S230 includes:

[0146] If there is no fire in the target area, all the electric control switches supplying power to the target area are controlled to be turned on.

[0147] It should be noted here that if it is determined that there is no fire in the target area, all the electric switches supplying power to the target area are controlled to be turned on, and the power supply to the target area is restored, which further improves the intelligence and usability of the present invention and greatly reduces the workload of relevant staff.

[0148] Specifically, if there is a target image defined as a target human body in the current frame image, outputting an alarm signal about the presence of the target human body in the suspected fire area also includes:

[0149] The voice playback device installed in the target area outputs a voice alarm signal about the suspected fire in the target area and the request to leave the target area immediately.

[0150] It should be noted here that if there is a target image defined as a target human body in the current frame image, a voice alarm signal (the alarm signal in the above embodiment) is output regarding the suspected fire in the target area and the person is requested to leave the target area immediately, thereby prompting the target person (the living object in the above embodiment) to leave the target area immediately, thereby making the accumulated value of the second counter greater than or equal to the third preset threshold as quickly as possible, and controlling all the electric switches supplying power to the target area to be disconnected as quickly as possible, further preventing the spread of electrical fires, and further improving the intelligence, reliability and usability of the present invention.

[0151] Specifically, if there is a target image defined as a target human body in the current frame image, outputting an alarm signal about the presence of the target human body in the suspected fire area also includes:

[0152] By means of image processing, the positioning coordinates of each target person are obtained, and the positioning data of each escape exit is obtained by using the positioning devices set at the escape exits of the flexible production line. Then, according to the position of the positioning coordinates of each target person in the environmental map, the optimal planning of the escape route is carried out, and it is determined whether there are more than two escape routes, and at the same time, it is determined whether the number of people on a certain escape path is greater than or equal to a fourth preset threshold. If so, any escapee on the escape path is selected, and it is determined whether the extension time of the escape time of the person is greater than or equal to the fifth preset threshold if the escape path of the person is changed to the path with the second shortest escape time for the person. If not, the escape path of the person is changed to the path with the second shortest escape time for the person, and the escape path is sent to the staff of the flexible production line through the cloud server.

[0153] Specifically, the method of obtaining the location coordinates of each target person by using image processing includes:

[0154] When the YOLOV5 algorithm is used to identify the image of a target person, a target search box for the target person is established. The target search box is a rectangle, and the rectangular target search box just completely covers the image of the target person. The pixel coordinates of the center point of the rectangular target search box are calculated, and the pixel coordinates of the center point of the rectangular target search box are marked as the pixel coordinates of the center point of the image of the target person. The pixel coordinates of the center point of the image of the target person are converted into geographic coordinates using the homography matrix, that is, the positioning coordinates of the target person are obtained.

[0155] Specifically, the present invention further provides a preset personnel movement speed in the control module, and the calculation method of the extension time of the personnel's escape time is the quotient of the extension length of the personnel's escape path after the change and the preset personnel movement speed.

[0156] Specifically, the optimal planning of the escape route is the route with the shortest escape time, that is, the path with the shortest length, when the moving speed of the personnel is constant.

[0157] It can be understood that the present invention realizes the detection of whether there is a suspected fire in the target area, or whether there is a fire, or whether there is a target human body, and performs data fusion analysis only by setting a smoke alarm in the target area of ​​the flexible production line and a camera above the target area, and then determines whether to control all the electric control switches supplying power to the target area to be disconnected, which solves the problem in the prior art that when a fire occurs in the target area of ​​the flexible production line, if the power supply to the target area is not cut off in time, the electrical fire may spread, and if the power supply is cut off when someone has not left the target area, the difficulty of escape may be increased. The hardware system is greatly simplified, and no complex algorithm modeling is required. Combined with the environment of the flexible production line, the environmental map of the flexible production line is updated in real time, and all escape paths of the flexible production line are planned in real time. The problem that the production equipment of the flexible production line is often not fixed, the position and type will change with the project plan, that is, the escape path may change, the production environment is relatively complex, and the difficulty of escape for relevant personnel is further increased is solved. The intelligence and usability of the present invention are greatly improved, the application scenarios of the present invention are greatly expanded, and new ideas are provided for intelligent safety monitoring of flexible production lines.

[0158] Figure 6 A schematic diagram of the composition structure of a production line control device based on data perception provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the production line control device 600 based on data perception includes: an acquisition unit 601, an identification module 602 and an execution module 603, wherein:

[0159] An acquisition module 601 is used to acquire current smoke concentration data of a target area of ​​a production line;

[0160] The recognition module 602 is used to determine the target object recognition result of the target area based on the video stream data of the target area when the current smoke concentration data meets the accident judgment condition; the target object recognition result is at least used to characterize whether there is a living object in the target area;

[0161] The execution module 603 is used to determine the accident execution strategy for the target area based on the target object recognition result of the target area, and execute the accident execution strategy for the target area; the accident execution strategy at least includes a strategy for controlling the operating state of the target area.

[0162] In some embodiments, the video stream data includes at least two frames of image data; the above-mentioned recognition module 602 is also used to perform living object recognition processing on the at least two frames of image data to obtain an image recognition result for each of the image data; the image recognition result is used to characterize whether there is a living object in the image data; based on the image recognition result of each of the image data, the target object recognition result of the target area is determined.

[0163] In some embodiments, the above-mentioned recognition module 602 is also used to determine a first number of target image data in which no living objects exist based on the image recognition results of each of the image data; when the first number is greater than or equal to a first threshold and the first number of target image data is continuous in image acquisition time, it is determined that there are no living objects in the target area; when the first number is less than the first threshold, it is determined that there are living objects in the target area.

[0164] In some embodiments, the execution module 603 is also used to determine a first strategy for controlling the operating state of the target area to be in a completely disconnected state when the target object recognition result indicates that there is no living object in the target area; or, when the target object recognition result indicates that there is a living object in the target area, determine a second strategy for outputting an alarm signal, determining an escape path for the living object in the target area, and controlling the operating state of the target area to be in a partially disconnected state.

[0165] In some embodiments, the execution module 603 is also used to obtain real-time map information of the production line; the real-time map information is determined based on the real-time attribute information of each production equipment of the production line and the coordinate range information of the production line; based on the image data of the living object, the position information of the living object in the real-time map information is determined; based on the position information of the living object and the real-time map information of the production line, at least one escape path of the living object is determined.

[0166] In some embodiments, the above-mentioned data-perception-based production line control device 600 also includes a path replacement module; the path replacement module is used to determine the number of living objects on each escape path based on the coordinate range information of the escape path and the position information of the living objects when the number of the escape paths is at least two; for a target escape path in which the number of living objects is greater than or equal to a second threshold, determine the escape extension time of the second number of living objects on the target escape path when the escape path is replaced; the difference between the number of living objects on the target escape path and the second number is less than the second threshold; when the escape extension time is less than a third threshold, replace the escape path of the living objects of the second data; the number of living objects on the replaced escape path is less than the second threshold.

[0167] In some embodiments, the above-mentioned identification module 602 is also used to determine the duration of the current smoke concentration data being greater than or equal to the fourth threshold when the current smoke concentration data is greater than or equal to the fourth threshold; and to determine that the current smoke concentration data meets the accident judgment condition when the duration is greater than or equal to the fifth threshold.

[0168] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0169] It should be noted that in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0170] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0171] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0172] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0173] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0174] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0175] Figure 7 is a schematic diagram of a hardware entity of an electronic device in an embodiment of the present application, such as Figure 7 As shown, the hardware entity of the electronic device 700 includes: a processor 701, a communication interface 702 and a memory 703, wherein:

[0176] The processor 701 generally controls the overall operation of the electronic device 700 , and the overall operation may be to implement the processing method provided in the embodiment of the present application.

[0177] The communication interface 702 enables the computer device to communicate with other terminals or servers through a network.

[0178] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or processed by the processor 701 and each module in the electronic device 700 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 701, the communication interface 702, and the memory 703 through the bus 704.

[0179] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the processing method of any of the above embodiments.

[0180] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0181] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0182] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0183] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0184] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0185] The above are only implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A production line control method based on data perception, the method comprising: Obtain current smoke concentration data in the target area of ​​the production line; In a case where the current smoke concentration data meets the accident judgment condition, determining a target object recognition result of the target area based on the video stream data of the target area; The target object recognition result is at least used to characterize whether there is a living object in the target area; Determine an accident execution strategy for the target area based on the target object recognition result of the target area, and execute the accident execution strategy for the target area; The accident execution strategy at least includes a strategy for controlling the operating state of the target area.

2. The method according to claim 1, wherein the video stream data comprises at least two frames of image data; and determining the target object recognition result of the target area based on the video stream data of the target area comprises: Performing target object recognition processing on the at least two frames of image data to obtain an image recognition result for each of the image data; The image recognition result is used to indicate whether there is a living object in the image data; Based on the image recognition result of each of the image data, a target object recognition result of the target area is determined.

3. The method according to claim 2, wherein the step of determining the target object recognition result of the target area based on the image recognition result of each of the image data comprises: determining a first number of target image data in which no living object exists based on the image recognition result of each of the image data; When the first number is greater than or equal to a first threshold value and the first number of target image data is continuous in image acquisition time, determining that there is no living object in the target area; When the first number is less than a first threshold, it is determined that a living object exists in the target area.

4. The method according to claim 1, wherein determining the accident execution strategy for the target area based on the target object recognition result of the target area comprises: In a case where the target object recognition result indicates that there is no living object in the target area, determining a first strategy for controlling the operation state of the target area to be in a fully disconnected state; Alternatively, when the target object recognition result indicates that there is a living object in the target area, a second strategy is determined to output an alarm signal, determine an escape path for the living object in the target area, and control the operating state of the target area to be in a partially disconnected state.

5. The method according to claim 4, wherein determining an escape path for the living object in the target area comprises: Obtaining real-time map information of the production line; The real-time map information is determined based on the real-time attribute information of each production equipment of the production line and the coordinate range information of the production line; Based on the image data containing the living object, determining the position information of the target object in the real-time map information; At least one escape path of the target object is determined based on the location information of the living object and the real-time map information of the production line.

6. The method according to claim 5, further comprising: When the number of the escape paths is at least two, determining the number of living objects on each of the escape paths based on the coordinate range information of the escape paths and the position information of the living objects; For a target escape path in which the number of living objects is greater than or equal to a second threshold, determining an escape extension time for a second number of living objects on the target escape path when the escape path is changed; The difference between the number of living objects on the target escape path and the second number is less than the second threshold; When the escape extension time is less than a third threshold, the escape path of the living object of the second data is changed; and the number of the living objects on the changed escape path is less than the second threshold.

7. According to any one of the methods of claims 1 to 6, whether the current smoke density data meets the accident judgment condition comprises: In a case where the current smoke density data is greater than or equal to a fourth threshold, determining a duration during which the current smoke density data is greater than or equal to the fourth threshold; When the duration is greater than or equal to the fifth threshold, it is determined that the current smoke density data meets the accident judgment condition.

8. A production line control device based on data perception, characterized in that: The device comprises: An acquisition module is used to obtain the current smoke concentration data of the target area of ​​the production line; an identification module, configured to determine a target object identification result of the target area based on the video stream data of the target area when the current smoke concentration data meets the accident judgment condition; the target object identification result is at least used to characterize whether there is a living object in the target area; An execution module is used to determine an accident execution strategy for the target area based on the target object recognition result of the target area, and execute the accident execution strategy for the target area; the accident execution strategy at least includes a strategy for controlling the operating state of the target area.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images