Production line abnormal situation detection method, system and terminal

Abstract

The application relates to a production line abnormality detection method and system and a terminal, and relates to the field of production line detection.The method comprises the following steps: acquiring image detection original data and laser detection original data of a production line; controlling a preset visual modal processing branch to extract a visual feature vector according to the image detection original data; controlling a preset laser modal processing branch to extract a laser feature vector according to the laser detection original data; inputting the visual feature vector and the laser feature vector into a preset cross-modal fusion module for feature fusion to generate a fusion feature matrix; inputting the fusion feature matrix into a preset classification large model for classification to determine a production line abnormality type; and prompting personnel according to the production line abnormality type.The application has the effects of improving the accuracy of production line abnormality detection and reducing the missed detection rate of production line abnormality.

Description

Technical Field

[0001] This application relates to the technical field of production line inspection, and in particular to a method, system and terminal for detecting abnormal conditions on a production line. Background Technology

[0002] Production line anomaly detection refers to the means of real-time monitoring, analysis, and handling of abnormal situations on the production line during the production process.

[0003] In related technologies, abnormal situations on production lines are typically detected using visual information or millimeter-wave radar information, such as whether workers have entered a specific area or whether workers have a risk conflict with production equipment.

[0004] Regarding the aforementioned technologies, which rely on visual information or millimeter-wave radar information for detection, these two types of data are processed independently and linked only by simple logic (such as threshold judgment). They cannot deeply integrate features, resulting in a high rate of missed detection of abnormal situations on the production line, and there is still room for improvement. Summary of the Invention

[0005] To improve the accuracy of production line anomaly detection and reduce the missed detection rate of production line anomalies, this application provides a production line anomaly detection method, system, and terminal.

[0006] Firstly, this application provides a method for detecting abnormal conditions on a production line, employing the following technical solution:

[0007] A method for detecting abnormal conditions on a production line, comprising:

[0008] Acquire raw image detection data and raw laser detection data from the production line;

[0009] Based on the raw image detection data, the preset visual modality processing branch is controlled to extract visual feature vectors;

[0010] Based on the raw data from laser detection, the preset laser mode processing branch is controlled to extract laser feature vectors;

[0011] The visual feature vector and the laser feature vector are input into a preset cross-modal fusion module for feature fusion to generate a fused feature matrix;

[0012] The fused feature matrix is ​​input into a pre-defined classification model for classification to determine the type of production line anomaly.

[0013] Provide alerts to personnel based on the type of production line abnormality.

[0014] By adopting the above technical solution, the visual modality processing branch is controlled to extract visual feature vectors based on the original image detection data, and the laser modality processing branch is controlled to extract laser feature vectors based on the original laser detection data. The visual feature vectors and laser feature vectors are then input into the cross-modal fusion module for deep feature fusion to obtain a fusion feature matrix. The classification model is then controlled to determine the type of production line anomaly based on the fusion feature matrix, thereby improving the accuracy of production line anomaly detection and reducing the false negative rate of production line anomalies.

[0015] Optionally, the steps for alerting personnel based on the type of production line anomaly include:

[0016] Determine whether the type of production line abnormality meets the requirements of the preset fire scenario type;

[0017] If the condition is not met, the personnel will be prompted according to the preset distance warning message;

[0018] If the conditions are met, obtain the location of the personnel and the location of the fire;

[0019] Based on the location of personnel and the location of the fire, assist personnel in leaving the production line.

[0020] By adopting the above technical solution, when the abnormality type of the production line does not meet the requirements of the fire scenario type, the system will prompt personnel to stay away from the area based on the principle prompt information. When it meets the requirements of the fire scenario type, the system will assist personnel in leaving the production line based on the personnel location and the fire location, thereby improving the protection of personnel when an abnormality occurs on the production line.

[0021] Optionally, steps to assist personnel in leaving the production line, based on their location and the location of the fire, include:

[0022] Obtain safe evacuation routes based on personnel locations;

[0023] Determine whether the location of the fire meets the requirements of a safe evacuation route;

[0024] If the conditions are met, the optimal evacuation route is obtained;

[0025] If not, obtain the fire situation of the safe evacuation route;

[0026] Analyze the fire situation and corresponding safe evacuation routes to determine the optimal evacuation route;

[0027] The system uses a pre-set auxiliary evacuation device based on the optimal evacuation path to assist personnel in leaving the production line.

[0028] By adopting the above technical solution, a safe evacuation route is determined based on the personnel's location. The optimal evacuation route is selected by comprehensively considering the fire location and fire situation. The optimal evacuation route is then used to control the auxiliary evacuation device to assist personnel in leaving the production line, thereby improving the safety of personnel when evacuating the production line.

[0029] Optionally, the steps for obtaining a safe evacuation route based on personnel location include:

[0030] The safety path is designed based on the personnel's location and the preset location path relationship;

[0031] Determine whether the personnel's location meets the preset requirements for the lower floor location;

[0032] If it does not meet the requirements, the designed safe path will be defined as a safe evacuation path;

[0033] If the match is found, obtain the window position;

[0034] Establish additional safety routes based on window location, personnel location, and fire location;

[0035] Associate the design of secure paths and additional secure paths to generate a secure evacuation route.

[0036] By adopting the above technical solution, the designed safety path is determined based on the personnel location and the relationship between the location path. When the personnel location meets the requirements of the lower floor location, an additional safety path is established based on the window location, personnel location and fire location. The designed safety path and the additional safety path are then associated to obtain a safe evacuation path, enabling personnel to leave the production line as quickly as possible, thereby improving the safety of personnel when evacuating from the production line.

[0037] Optionally, the steps for assisting personnel to leave the production line by controlling a preset auxiliary evacuation device according to the optimal evacuation path include:

[0038] The parameters for starting the device are determined based on the personnel's location and the preset relationship between the device and the location.

[0039] The auxiliary evacuation device is activated according to the activation device parameters, and the auxiliary evacuation device is controlled to move to the personnel's location;

[0040] The auxiliary evacuation device guides personnel away from the production line based on the optimal evacuation path and obtains the real-time evacuation location.

[0041] Analyze real-time evacuation locations and optimal evacuation routes to determine the deployment path;

[0042] The evacuation device uses a projection path control system to project a path to guide personnel away from the production line.

[0043] By adopting the above technical solution, the auxiliary evacuation device is activated according to the activation device parameters, so that the auxiliary evacuation device can reach the personnel position as quickly as possible, guide the personnel away from the production line according to the optimal evacuation path, and project the projection path to guide the personnel, thereby improving the reliability of assisting personnel to evacuate from the production line.

[0044] Optionally, the steps for controlling the auxiliary evacuation device to guide personnel away from the production line according to the optimal evacuation path include:

[0045] The control auxiliary evacuation device guides personnel away from the production line along the optimal evacuation path and acquires environmental parameters of the optimal evacuation path in real time.

[0046] Determine whether the environmental parameters meet the preset requirements for hazard parameters;

[0047] If it does not meet the requirements, the environmental parameters of the optimal evacuation route will be continuously obtained in real time for iterative judgment.

[0048] If the conditions are met, the auxiliary evacuation device will send a preset cooling trigger signal.

[0049] The system controls the preset fire-fighting devices to spray water mist to achieve cooling based on the cooling trigger signal, and obtains the fire response status.

[0050] The auxiliary evacuation device is controlled to assist in cooling based on the fire response situation.

[0051] By adopting the above technical solution, when the environmental parameters meet the requirements of the hazardous parameters, the auxiliary evacuation device is controlled to send a cooling trigger signal, so that the fire-fighting device on the optimal evacuation route receives the cooling trigger signal and sprays water mist to achieve cooling, thereby reducing the impact of high temperature on personnel during the evacuation of the production line.

[0052] Optionally, the steps for controlling the auxiliary evacuation device to assist in cooling, based on the fire response situation, include:

[0053] Determine whether the fire response status meets the preset requirements for a responded status;

[0054] If the conditions are met, continue to obtain fire response information and perform a cyclical judgment.

[0055] If the conditions are not met, the auxiliary evacuation device will spray water mist above the personnel to cool them down.

[0056] By adopting the above technical solution, when it is determined that the fire response does not meet the requirements of the response, the auxiliary evacuation device is controlled to spray water mist above the personnel, thereby ensuring that in areas without fire-fighting equipment or with damaged fire-fighting equipment, the auxiliary evacuation device can replace the fire-fighting equipment for cooling, thereby improving the safety of personnel during evacuation.

[0057] Optionally, the step of controlling the auxiliary evacuation device to spray water mist above personnel to achieve cooling includes:

[0058] Obtain information on support personnel;

[0059] Determine whether the assistant's status is a preset single-person assistant status or a preset multi-person assistant status;

[0060] If it is in single-person auxiliary mode, the auxiliary evacuation device will directly spray water mist to achieve cooling according to the preset single-person spray parameters.

[0061] If it is a multi-person assistance state, then obtain the center location and range of the multi-person team;

[0062] Analyze the area with multiple users to determine the spray range;

[0063] The auxiliary evacuation device is controlled to move to the center of the multi-person area to guide personnel away from the production line, and the auxiliary evacuation device sprays water mist according to the spray range to achieve cooling.

[0064] By adopting the above technical solution, when the situation of the auxiliary personnel is determined to be a single-person auxiliary state, the auxiliary evacuation device is directly controlled to spray water mist according to the single-person spray parameters; when the situation of the auxiliary personnel is a multi-person auxiliary state, the spray range is determined according to the range of the multi-person, so as to control the auxiliary evacuation device to move to the center position of the multi-person and spray water mist within the spray range to ensure that the multi-person is cooled at the same time, thereby improving the safety of personnel during evacuation.

[0065] Secondly, this application provides a production line abnormality detection system, which adopts the following technical solution:

[0066] A production line anomaly detection system, comprising:

[0067] The acquisition module is used to acquire raw image detection data and raw laser detection data;

[0068] A memory for storing a program for a production line anomaly detection method as described in any of the preceding claims;

[0069] The processor and the program in the memory can be loaded and executed by the processor to implement a production line abnormality detection method as described in any of the above.

[0070] By adopting the above technical solution, the processor loads and executes a program for a production line anomaly detection method stored in the memory. This controls the acquisition module to acquire a series of data related to production line anomaly detection, controls the visual modality processing branch to extract visual feature vectors based on the raw image detection data, and controls the laser modality processing branch to extract laser feature vectors based on the raw laser detection data. The visual feature vectors and laser feature vectors are then input into the cross-modal fusion module for deep feature fusion to obtain a fusion feature matrix. Finally, the classification model is controlled to determine the type of production line anomaly based on the fusion feature matrix, thereby improving the accuracy of production line anomaly detection and reducing the false negative rate of production line anomalies.

[0071] Thirdly, this application provides a smart terminal, which adopts the following technical solution:

[0072] A smart terminal includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any of the preceding claims for detecting production line anomalies.

[0073] By adopting the above technical solution, the processor of the intelligent terminal loads and executes a computer program for a production line anomaly detection method stored in the memory. This program controls the visual modality processing branch to extract visual feature vectors from the raw image detection data, and controls the laser modality processing branch to extract laser feature vectors from the raw laser detection data. The visual feature vectors and laser feature vectors are then input into the cross-modal fusion module for deep feature fusion to obtain a fusion feature matrix. Finally, the classification model is controlled to determine the type of production line anomaly based on the fusion feature matrix, thereby improving the accuracy of production line anomaly detection and reducing the false negative rate of production line anomalies.

[0074] In summary, this application includes at least one of the following beneficial technical effects:

[0075] 1. By controlling the visual modality processing branch to extract visual feature vectors from the original image detection data, and controlling the laser modality processing branch to extract laser feature vectors from the original laser detection data, the visual feature vectors and laser feature vectors are input into the cross-modal fusion module for deep feature fusion to obtain a fusion feature matrix. Then, the classification model is controlled to determine the type of production line anomaly based on the fusion feature matrix, thereby improving the accuracy of production line anomaly detection and reducing the false negative rate of production line anomalies.

[0076] 2. By activating the auxiliary evacuation device according to the activation device parameters, the auxiliary evacuation device can reach the personnel location as quickly as possible, guide the personnel out of the production line according to the optimal evacuation path, and project the projection path to guide the personnel, thereby improving the reliability of assisting personnel to evacuate from the production line.

[0077] 3. When the auxiliary personnel are in a single-person auxiliary state, the auxiliary evacuation device is directly sprayed with water mist based on the single-person spray parameters; when the auxiliary personnel are in a multi-person auxiliary state, the spray range is determined based on the range of the multi-person auxiliary personnel, and the auxiliary evacuation device is moved to the center of the multi-person auxiliary personnel and sprayed with water mist within the spray range to ensure that the multi-person auxiliary personnel are cooled at the same time, thereby improving the safety of personnel during evacuation. Attached Figure Description

[0078] Figure 1 This is a flowchart of a production line abnormality detection method in an embodiment of this application.

[0079] Figure 2 This is a flowchart of the steps for alerting personnel based on the type of production line abnormality in this embodiment of the application.

[0080] Figure 3 This is a flowchart of the steps in this application embodiment to assist personnel in leaving the production line based on their location and the location of the fire.

[0081] Figure 4 This is a flowchart of the steps for obtaining a safe evacuation route based on the location of personnel in an embodiment of this application.

[0082] Figure 5 This is a flowchart of the steps in this application embodiment to control a preset auxiliary evacuation device to assist personnel in leaving the production line according to the optimal evacuation path.

[0083] Figure 6 This is a flowchart of the steps in this application embodiment to control the auxiliary evacuation device to guide personnel away from the production line according to the optimal evacuation path.

[0084] Figure 7 This is a flowchart of the steps in this application embodiment to control the auxiliary evacuation device to assist in cooling based on the fire response situation.

[0085] Figure 8 This is a flowchart of the steps in this application embodiment of controlling the auxiliary evacuation device to spray water mist above personnel to achieve cooling. Detailed Implementation

[0086] To make the purpose, technical solution, and advantages of this application clearer, the following description is provided in conjunction with the appendix. Figures 1 to 8 The present application will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the application.

[0087] This application discloses a method for detecting production line anomalies. After relevant sensors collect raw image detection data and raw laser detection data, a visual modality processing branch analyzes and processes the raw image detection data to extract visual feature vectors, and a laser modality processing branch analyzes and processes the raw laser detection data to extract laser feature vectors. Then, a cross-modal fusion module is controlled to fuse the visual feature vectors and laser feature vectors to obtain a fused feature matrix. The classification model identifies production line anomalies based on the fused feature matrix, obtains the type of production line anomaly, and provides a prompt, thereby improving the accuracy of production line anomaly detection and reducing the false negative rate of production line anomalies.

[0088] Reference Figure 1 This application discloses a method for detecting abnormal conditions on a production line, comprising the following steps:

[0089] Step S100: Obtain raw image detection data and raw laser detection data from the production line.

[0090] The image detection raw data refers to image data within the production line area, including 2D images, video streams, and depth maps. This data can be captured by cameras installed within the production line and preprocessed through distortion correction, white balance, and histogram equalization. The laser detection raw data refers to data scanned by lasers within the production line area, including point cloud data. Each point contains coordinates, reflection intensity, and a timestamp, as well as Doppler velocity (e.g., target radial velocity) and vibration spectrum. Frequency domain features are extracted through FFT analysis, and this data can be detected by a solid-state LiDAR. The detection of both image and laser detection raw data requires spatiotemporal alignment through hardware synchronization and spatial calibration. GPS / PPS (pulses per second) signals or hardware trigger lines are used to synchronize the timestamps of the cameras and LiDAR. Spatial calibration employs a checkerboard calibration method.

[0091] Step S101: Extract visual feature vectors by controlling the preset visual modality processing branch based on the raw image detection data.

[0092] The visual modality processing branch refers to the model used to extract spatial information of people and objects in an image. In this embodiment, YOLOv7 is used.

[0093] Visual feature vectors refer to the extracted target category, location coordinates, and pose vectors. Target detection uses an anchor-based mechanism to locate the bounding boxes of people and objects. Pose estimation is based on keypoint detection networks (such as HRNet branches) to analyze human skeletal joints. Temporal modeling uses optical flow or 3D convolution to track motion trajectories in video streams.

[0094] Step S102: Extract laser feature vectors by controlling the preset laser mode processing branch based on the original laser detection data.

[0095] Among them, the laser mode processing branch refers to the model used to extract distance, velocity and vibration signals from laser data. In this embodiment, a LiDAR point cloud network is used.

[0096] Laser feature vectors refer to the extracted target distance, velocity, and vibration spectrum. First, point cloud preprocessing is performed, and VoxelNet or PointPillars is used to voxelize the point cloud. Then, spatial distribution features are extracted through 3D convolution kernels. The radial velocity of the moving target is separated through FFT spectrum analysis, and wavelet transform is applied to extract the nanometer-level vibration frequency of the equipment surface.

[0097] Step S103: Input the visual feature vector and the laser feature vector into the preset cross-modal fusion module for feature fusion to generate a fused feature matrix.

[0098] The cross-modal fusion module refers to the module that performs feature-level fusion, including spatiotemporal alignment, attention interaction, and feature enhancement.

[0099] The fusion feature matrix refers to the feature matrix obtained by feature-level fusion of visual feature vectors and laser feature vectors. First, based on the hardware synchronization signal and calibration extrinsic parameter matrix, the spatiotemporal coordinates of the two modal features are mapped. Then, a cross-modal Transformer encoder is used to calculate attention weights with visual features as queries and laser features as keys / values ​​to generate the fusion feature matrix. Finally, the spatiotemporal correlation of the fusion features is modeled through a 4-layer multi-head self-attention layer, and position encoding is added to enhance spatial sensitivity. For example, when a person is detected crossing the boundary, the visual pose features are automatically associated with the instantaneous velocity features of the LiDAR.

[0100] Step S104: Input the fused feature matrix into the preset classification model for classification to determine the type of production line anomaly.

[0101] Among them, the classification model refers to the model that classifies production line anomalies based on the fused feature matrix. In this embodiment, a multi-task BERT variant model is used, which is jointly trained for classification and regression tasks through an uncertainty-weighted loss function.

[0102] Production line anomaly type refers to the type of anomaly that occurs on the production line. The fused feature matrix is ​​input into the classification model, and anomaly classification is performed simultaneously: the fully connected layer and the Softmax function output the category probability of personnel intrusion, equipment vibration exceeding limits, and other extreme anomalies; vibration regression: the L1 loss function is used to predict the micron-level value of vibration amplitude; positioning correction: the target position is compensated for error based on Kalman filter prior, and the calibrated three-dimensional coordinates are output.

[0103] Step S105: Provide prompts to personnel based on the type of production line abnormality.

[0104] After obtaining the type of production line anomaly, personnel are given a voice announcement or assisted to move away from the isolation area or production line, based on the type of anomaly. Specific methods are detailed below. Figure 2 These steps ensure the safety of personnel in the event of abnormal situations on the production line.

[0105] Reference Figure 2 The steps for alerting personnel based on the type of production line anomaly include:

[0106] Step S200: Determine whether the production line abnormality type meets the requirements of the preset fire scenario type.

[0107] Among them, the fire scenario type refers to the abnormal situation of the production line that results in a fire scenario, and the requirement for the fire scenario type is that it must be consistent with the fire scenario type.

[0108] By processing the terminal, it can determine whether the production line abnormality type is consistent with the fire scenario type, thereby determining whether a fire has occurred on the production line.

[0109] Step S201: If the condition is not met, prompt the personnel according to the preset distance prompt information.

[0110] If the processing terminal determines that the production line abnormality type is inconsistent with the fire scenario type, it indicates that the production line abnormality type is personnel entering the isolation area or equipment vibration exceeding the limit. Therefore, personnel can be prompted to stay away from the isolation area or equipment according to the stay-away prompt information.

[0111] "Away from equipment" messages are voice messages that remind personnel to stay away from equipment and isolated areas. These messages are stored in the processing terminal by the operator.

[0112] Step S202: If the conditions are met, obtain the personnel location and fire location.

[0113] If the processing terminal determines that the production line anomaly type is consistent with the fire scenario type, it indicates that a fire has occurred on the production line. At this time, personnel need to leave the production line. Therefore, the location of personnel and the location of the fire are detected to provide data support for subsequent assistance in leaving the production line.

[0114] Personnel location refers to the position of personnel on the production line, and fire location refers to the location where a fire occurred on the production line. The target location coordinates are obtained by the visual modality processing branch based on the target features of personnel and fire.

[0115] Step S203: Assist personnel to leave the production line based on their location and the location of the fire.

[0116] After determining the locations of personnel and the fire, assist personnel in leaving the production line based on their locations. Specific methods are detailed below. Figure 3 These steps ensure the safety of personnel in the event of production line malfunctions.

[0117] Reference Figure 3 The steps to assist personnel in leaving the production line based on their location and the location of the fire include:

[0118] Step S300: Obtain a safe evacuation route based on the personnel's location.

[0119] The safe evacuation route refers to the safe route for personnel to evacuate from the production line under normal circumstances. For specific methods of obtaining this route, please refer to [link / reference needed]. Figure 3 The steps.

[0120] Step S301: Determine whether the location of the fire meets the requirements of a safe evacuation route.

[0121] The requirement for a safe evacuation route is that it must be located outside of a designated safe evacuation route. The processing terminal determines whether the fire location is outside of a safe evacuation route, thereby determining whether the safe evacuation route is usable.

[0122] Step S3011: If the conditions are met, obtain the optimal evacuation route.

[0123] If the processing terminal determines that the fire location is outside the safe evacuation path, it indicates that the safe evacuation path is available. Therefore, the optimal evacuation path is invoked to provide data support for subsequent auxiliary personnel to safely move away from the production line.

[0124] The optimal evacuation route is the shortest safe evacuation route that is free from fire. The processing terminal selects the safe evacuation routes free from fire, compares the lengths of these routes, and selects the shortest route as the optimal evacuation route.

[0125] Step S3012: If not, obtain the fire situation of the safe evacuation route.

[0126] If the processing terminal determines that the fire location is within the safe evacuation route, it indicates that a fire has occurred within the safe evacuation route. Therefore, the fire situation is detected to provide data support for determining the optimal evacuation route.

[0127] The fire situation refers to the size of the fire, which is determined by extracting the fire area from the visual modality processing branch.

[0128] Step S302: Analyze the fire situation and the corresponding safe evacuation routes to determine the optimal evacuation route.

[0129] In this step, the optimal evacuation route is the same as the optimal evacuation route in step S3011. The processing terminal compares the fire situation in the safe evacuation routes and selects the route with the least fire situation and the shortest path as the optimal evacuation route.

[0130] Step S303: Control the preset auxiliary evacuation device according to the optimal evacuation path to assist personnel in leaving the production line.

[0131] After determining the optimal evacuation route, the terminal control auxiliary evacuation device assists personnel in leaving the production line. Specific methods are detailed in [reference needed]. Figure 5 These steps ensure the safety of personnel evacuation.

[0132] An evacuation assistance device is a device that helps personnel leave the production line. It can be a drone that guides personnel out of the production line. The drone carries a spray device to cool down the area and a projection device to project the path onto the ground to guide the personnel.

[0133] Reference Figure 4 The steps for obtaining a safe evacuation route based on personnel location include:

[0134] Step S400: Determine the design safety path based on the personnel location and the preset location path relationship.

[0135] Among them, the location path relationship refers to the correspondence between different locations and safety paths. The operator forms a mapping table by matching different safety paths with the locations on the path one by one.

[0136] Designing a safe path refers to the safe path designed in the production line, which is obtained by the processing terminal by looking up the corresponding mapping table of location and path relationship based on the personnel's location.

[0137] Step S401: Determine whether the personnel's position meets the preset requirements for the lower level position.

[0138] Among them, the lower level position refers to the position of personnel located on the first level, and the requirement for the lower level position is that it is consistent with the lower level position.

[0139] The system determines whether a person is on the first floor by processing the terminal to see if their location matches the location on the lower floor.

[0140] Step S4011: If it does not meet the requirements, then define the designed safe path as a safe evacuation path.

[0141] If the processing terminal determines that the location of the personnel is inconsistent with the location of the lower floor, it indicates that the personnel are at a higher altitude and the window cannot be used as an evacuation route. Therefore, the safety path is defined as the safe evacuation path.

[0142] Step S4012: If the condition is met, obtain the window position.

[0143] If the processing terminal determines that the personnel's location is consistent with the location on the lower floor, it indicates that the personnel are on the first floor and the window can be used as an evacuation route. Therefore, the window location is called to provide data support for determining the safe evacuation route.

[0144] Window location refers to the location of windows in the production line area. Operators upload and store the window locations of the production line area in the processing terminal.

[0145] Step S402: Establish additional safety paths based on window location, personnel location, and fire location.

[0146] Among them, the additional safety path refers to the path established using windows as evacuation routes. The processing terminal establishes evacuation paths based on the window positions and personnel positions, and then selects the evacuation paths that do not coincide with the fire location as additional safety paths.

[0147] Step S403: Associate the designed safe path and additional safe paths to generate a safe evacuation path.

[0148] In this step, the safe evacuation path is the same as the safe evacuation path in step S300. The processing terminal stores the designed safe path and the additional safe path in the same data packet.

[0149] Reference Figure 5 The steps for assisting personnel to leave the production line by controlling the preset auxiliary evacuation device according to the optimal evacuation path include:

[0150] Step S500: Determine the starting device parameters based on the personnel position and the preset position device relationship.

[0151] The location device relationship refers to the correspondence between personnel location and the nearest auxiliary evacuation device. The operator forms a mapping table by matching the location of the auxiliary evacuation device with the location that the device can assist, so that each auxiliary evacuation device assists all locations within a region.

[0152] The activation device parameters refer to the auxiliary evacuation devices that need to be activated, which are obtained by the processing terminal by looking up the corresponding mapping table of location devices based on the personnel's location.

[0153] Step S501: Activate the auxiliary evacuation device according to the activation device parameters, and control the auxiliary evacuation device to move to the personnel position.

[0154] After determining the parameters of the starting device, the processing terminal starts the corresponding auxiliary evacuation device according to the parameters and controls the auxiliary evacuation device to move to the personnel's location, thereby preparing for the subsequent evacuation of personnel from the production line.

[0155] Step S502: Control the auxiliary evacuation device to guide personnel away from the production line according to the optimal evacuation path, and obtain the real-time evacuation location.

[0156] Once the auxiliary evacuation device reaches the personnel's location, it guides them away from the production line along the optimal evacuation path. The specific method is described in [reference needed]. Figure 6 The steps are followed, and the real-time evacuation location is detected to provide data support for subsequent guidance.

[0157] Real-time evacuation location refers to the real-time location of the auxiliary evacuation device during the evacuation process. The location coordinates can be obtained by extracting the image of the auxiliary evacuation device from the visual modality processing branch.

[0158] Step S503: Analyze the real-time evacuation location and the optimal evacuation path to determine the projection path.

[0159] The projection path refers to the portion of the path projected onto the ground, which is obtained by the processing terminal based on the real-time evacuation location's position on the optimal evacuation path, thereby identifying the next part of the evacuation path from the real-time evacuation location.

[0160] Step S504: Control the auxiliary evacuation device to project a path according to the projection path to guide personnel away from the production line.

[0161] After the projection path is determined, the projection device in the auxiliary evacuation device projects the projection path onto the ground, so that personnel can see the exit path more clearly during the process of leaving the production line, ensuring the accuracy and speed of the evacuation.

[0162] Reference Figure 6 The steps for using the assisted evacuation device to guide personnel out of the production line according to the optimal evacuation path include:

[0163] Step S600: Control the auxiliary evacuation device to guide personnel away from the production line along the optimal evacuation path, and obtain the environmental parameters of the optimal evacuation path in real time.

[0164] After the auxiliary evacuation device reaches the personnel's location, the processing terminal controls the auxiliary evacuation device to guide the personnel away from the production line along the optimal evacuation path, and detects the environmental parameters of the optimal evacuation path to provide data support for determining whether fire intervention is needed.

[0165] Environmental parameters refer to environmental parameters related to the optimal evacuation route, including temperature and flame conditions. Temperature is detected by temperature sensors on the assisted evacuation device, and flame conditions are obtained by feature extraction from images acquired by the assisted evacuation device by the visual modality processing branch.

[0166] Step S601: Determine whether the environmental parameters meet the preset requirements for hazardous parameters.

[0167] Hazard parameters refer to environmental parameters where the temperature is too high or flames are present along the evacuation route, and the requirements for hazard parameters are consistent with the conditions corresponding to the hazard parameters.

[0168] By processing the terminal, it is determined whether the environmental parameters correspond to the conditions corresponding to the hazard parameters, thereby determining whether fire intervention is required on the optimal evacuation route.

[0169] Step S6011: If it does not meet the requirements, continue to obtain the environmental parameters of the optimal evacuation path in real time and perform iterative judgment.

[0170] If the processing terminal determines that the environmental parameters are inconsistent with the hazardous parameters, it indicates that the temperature on the optimal evacuation route is low and there is no flame. Therefore, no fire intervention is required. The environmental parameters of the optimal evacuation route will continue to be monitored to keep track of environmental changes along the optimal evacuation route.

[0171] Step S6012: If the condition is met, the auxiliary evacuation device is controlled to issue a preset cooling trigger signal.

[0172] If the processing terminal determines that the environmental parameters correspond to the same situation as the hazard parameters, it indicates that there is a high temperature or flame on the optimal evacuation route. Therefore, fire intervention is required, which will cause the auxiliary evacuation device to send a cooling trigger signal to control the fire-fighting device to spray water mist.

[0173] The cooling trigger signal is a signal that controls the water mist spraying of the fire-fighting device. It is sent by the auxiliary evacuation device and received and responded to by the fire-fighting device.

[0174] Step S602: Control the preset fire-fighting device to spray water mist to achieve cooling according to the cooling trigger signal, and obtain the fire response status.

[0175] The fire-fighting device receives and responds to a cooling trigger signal to spray water mist, thereby achieving cooling along the optimal evacuation route, ensuring the safety of personnel evacuation, and monitoring the fire response to provide data support for determining whether auxiliary evacuation devices are needed for subsequent spraying.

[0176] Fire suppression equipment refers to fire sprinklers installed within the production line for fire extinguishing. Fire response status refers to the response status of the fire suppression equipment, including whether it has responded or not. The fire suppression equipment sends a feedback signal to the processing terminal. If the processing terminal receives the feedback signal, it determines that it has responded; otherwise, it determines that it has not responded.

[0177] Step S603: Control the auxiliary evacuation device to perform auxiliary cooling according to the fire response situation.

[0178] Upon receiving a fire response, the processing terminal controls the auxiliary evacuation device to perform auxiliary cooling based on the fire response information. The specific method is as follows: Figure 7 This process ensures that when the fire-fighting equipment fails to respond, the auxiliary evacuation device can promptly replace the fire-fighting equipment for cooling.

[0179] Reference Figure 7 The steps for controlling the auxiliary evacuation device to assist in cooling based on the fire response include:

[0180] Step S700: Determine whether the fire response status meets the preset requirements for the responded status.

[0181] Among them, "responded status" refers to the fire-fighting device responding to the fire-fighting trigger signal and spraying water mist normally. The requirement for "responded status" is that it is consistent with the "responded status".

[0182] By processing the terminal, it is determined whether the fire response status is consistent with the already responded status, thereby determining whether the fire protection device is working properly.

[0183] Step S701: If the condition is met, continue to obtain the fire response information and perform a cyclical judgment.

[0184] If the processing terminal determines that the fire response status is consistent with the already responded status, it indicates that the fire protection device is working normally. Therefore, the fire response status will continue to be monitored to keep track of the working status of the fire protection device.

[0185] Step S702: If not, control the auxiliary evacuation device to spray water mist above the personnel to achieve cooling.

[0186] If the processing terminal determines that the fire response status is inconsistent with the actual response status, it indicates that the fire-fighting equipment cannot spray water mist. Therefore, the auxiliary evacuation device is controlled to spray water mist above the personnel, thereby replacing the fire-fighting equipment for cooling. Specific methods are detailed below. Figure 8 The steps.

[0187] Reference Figure 8 The steps for controlling the auxiliary evacuation device to spray water mist above personnel to achieve cooling include:

[0188] Step S800: Obtain information on auxiliary personnel.

[0189] Among them, the number of auxiliary personnel refers to the number of personnel assisted in the evacuation by the auxiliary evacuation device, which is obtained by the visual modality processing branch by extracting features from the images collected by the auxiliary evacuation device.

[0190] Step S801: Determine whether the assistant's status is a preset single-person assistant status or a preset multi-person assistant status.

[0191] Among them, the single-person assistance mode means that the evacuation assistance device assists only one person in evacuation, while the multi-person assistance mode means that the evacuation assistance device assists multiple people in evacuation.

[0192] The processing terminal determines whether the assistant is in a single-person or multi-person assistance state, and thus determines how to spray water mist.

[0193] Step S8011: If it is a single-person auxiliary state, the auxiliary evacuation device is controlled to spray water mist directly according to the preset single-person spray parameters to achieve cooling.

[0194] If the processing terminal determines that the auxiliary personnel are in a single-person auxiliary state, it means that the auxiliary evacuation device is only assisting one person in evacuation. Therefore, the auxiliary evacuation device is controlled to spray water mist above the personnel according to the single-person spraying parameters to achieve cooling.

[0195] The single-person spray parameter refers to the range of water mist sprayed by the auxiliary evacuation device on a single person. The specific value is determined by the operator based on the actual situation.

[0196] Step S8012: If it is a multi-person assistance state, then obtain the center position and range of the multi-person.

[0197] If the processing terminal determines that the auxiliary personnel are in a multi-person evacuation state, it indicates that the auxiliary evacuation device is assisting multiple people in evacuation at the same time. Therefore, the center position and range of the multiple people are detected to provide data support for the subsequent control of the auxiliary evacuation device to spray water mist.

[0198] The center position of multiple people refers to the middle position of multiple people, and the range of multiple people refers to the radius of the range of people. It is obtained by the visual modality processing branch by extracting features from the images collected by the auxiliary evacuation device.

[0199] Step S802: Analyze the area with multiple people to determine the spray range.

[0200] The spray range refers to the radius of the water mist sprayed by the auxiliary evacuation device, which is the radius corresponding to the area with multiple people.

[0201] Step S803: Control the auxiliary evacuation device to move to the center of the multi-person center to guide personnel away from the production line, and control the auxiliary evacuation device to spray water mist according to the spray range to achieve cooling.

[0202] After determining the location of the multi-person center, the processing terminal controls the auxiliary evacuation device to move to the multi-person center location to guide the personnel away from the production line, and controls the auxiliary evacuation device to spray water mist in the multi-person center location within the spray range, thereby effectively cooling everyone.

[0203] Based on the same inventive concept, embodiments of this application provide a production line anomaly detection system, including:

[0204] The acquisition module is used to acquire raw image detection data, raw laser detection data, personnel location, fire location, safe evacuation route, optimal evacuation route, fire situation, window location, real-time evacuation location, environmental parameters, fire response status, auxiliary personnel status, multi-person center location, and multi-person range.

[0205] A memory used to store a program for a production line anomaly detection method;

[0206] The processor can load and execute programs in memory to implement a method for detecting abnormal conditions on the production line.

[0207] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0208] This application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as a method for detecting abnormal conditions on a production line.

[0209] Computer storage media include, for example, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media that can store program code.

[0210] Based on the same inventive concept, this application provides a smart terminal, including a memory and a processor, wherein the memory stores a computer program that can be loaded and executed by the processor to perform a production line abnormality detection method.

[0211] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0212] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A method of detecting an abnormal situation in a production line, characterized by, The method comprises the following steps: acquiring image detection raw data and laser detection raw data of a production line; controlling a preset visual modal processing branch to extract a visual feature vector according to the image detection raw data; controlling a preset laser modal processing branch to extract a laser feature vector according to the laser detection raw data; inputting the visual feature vector and the laser feature vector into a preset cross-modal fusion module for feature fusion to generate a fusion feature matrix; inputting the fusion feature matrix into a preset classification large model for classification to determine a production line abnormality type; prompting personnel according to the production line abnormality type; the step of prompting personnel according to the production line abnormality type comprises: determining whether the production line abnormality type meets the requirements of a preset fire scene type; if not, prompting personnel according to preset away prompt information; if yes, acquiring personnel location and fire location; assisting personnel to leave the production line according to the personnel location and the fire location; the step of assisting personnel to leave the production line according to the personnel location and the fire location comprises: acquiring a safe evacuation path corresponding to the personnel location; determining whether the fire location meets the requirements of the safe evacuation path; if yes, acquiring a best evacuation path; if not, acquiring fire conditions of the safe evacuation path; analyzing the fire conditions and the corresponding safe evacuation path to determine the best evacuation path; controlling a preset auxiliary evacuation device to assist personnel to leave the production line according to the best evacuation path; the step of controlling the preset auxiliary evacuation device to assist personnel to leave the production line according to the best evacuation path comprises: determining starting device parameters according to the personnel location and a preset position device relationship; starting the auxiliary evacuation device according to the starting device parameters and controlling the auxiliary evacuation device to move to the personnel location; controlling the auxiliary evacuation device to lead personnel to leave the production line according to the best evacuation path and acquiring a real-time evacuation position; analyzing the real-time evacuation position and the best evacuation path to determine a projection path; controlling the auxiliary evacuation device to project the projection path to lead personnel to leave the production line according to the projection path; the step of controlling the auxiliary evacuation device to lead personnel to leave the production line according to the best evacuation path comprises: controlling the auxiliary evacuation device to lead personnel to leave the production line along the best evacuation path and acquiring environmental parameters of the best evacuation path in real time; determining whether the environmental parameters meet the requirements of a preset dangerous parameter; if not, continuously acquiring the environmental parameters of the best evacuation path in real time for cyclic determination; if yes, controlling the auxiliary evacuation device to send a preset cooling trigger signal; controlling a preset fire-fighting device to spray water mist to achieve cooling according to the cooling trigger signal and acquiring a fire-fighting response condition; controlling the auxiliary evacuation device to assist in cooling according to the fire-fighting response condition; the step of controlling the auxiliary evacuation device to assist in cooling according to the fire-fighting response condition comprises: determining whether the fire-fighting response condition meets the requirements of a preset responded condition; if yes, continuously acquiring the fire-fighting response condition for cyclic determination; if not, controlling the auxiliary evacuation device to spray water mist above the personnel to achieve cooling; the step of controlling the auxiliary evacuation device to spray water mist above the personnel to achieve cooling comprises: acquiring auxiliary personnel conditions; judging the assistant condition as a preset single-person assistant state or a preset multi-person assistant state; if the single-person assistant state, controlling the assistant evacuation device to directly spray water mist according to preset single-person spraying parameters to achieve cooling; if the multi-person assistant state, obtaining a multi-person center position and a multi-person range; analyzing the multi-person range to determine a spraying range; controlling the assistant evacuation device to move to the multi-person center position to lead the personnel to leave the production line, and controlling the assistant evacuation device to spray water mist according to the spraying range to achieve cooling.

2. The method of claim 1, wherein, The step of obtaining the safety evacuation path based on the personnel position comprises: determining a designed safety path according to the personnel position and a preset position-path relationship; judging whether the personnel position meets a preset low-layer position requirement; if not, defining the designed safety path as the safety evacuation path; if yes, obtaining a window position; establishing an additional safety path according to the window position, the personnel position and a fire position; associating the designed safety path and the additional safety path to generate the safety evacuation path.

3. A line abnormality detection system characterized by comprising: comprise: an acquisition module, configured to acquire image detection raw data and laser detection raw data; a memory, configured to store a program of the production line abnormality detection method according to any one of claims 1 to 2; a processor, the program in the memory being capable of being loaded and executed by the processor and realizing the production line abnormality detection method according to any one of claims 1 to 2.

4. A smart terminal, characterized by comprise a memory and a processor, the memory storing a computer program capable of being loaded and executed by the processor and realizing the production line abnormality detection method according to any one of claims 1 to 2.

Images