Production line abnormal condition detection method, system and terminal

Abstract

The invention relates to a production line abnormal condition detection method and system and a terminal, and relates to the field of production line detection, and the method comprises the steps: obtaining image detection original data and laser detection original data of a production line; according to the image detection original data, controlling a preset visual modal processing branch to extract a visual feature vector; controlling a preset laser modal processing branch to extract a laser feature vector according to the laser detection original data; inputting the visual feature vector and the laser feature vector into a preset cross-modal fusion module for feature fusion to generate a fusion feature matrix; inputting the fusion feature matrix into a preset classification large model for classification so as to determine a production line anomaly type; and prompting the personnel according to the abnormal type of the production line. The method and the device have the effects of improving the accuracy of abnormal condition detection of the production line and reducing the omission ratio of the abnormal conditions of the production line.

Description

Technical Field

[0001] The present application relates to the technical field of production line detection, and in particular to a method, system and terminal for detecting abnormal conditions in a production line. Background Art

[0002] Production line abnormality detection refers to the means of real-time monitoring, analysis and processing of production line abnormalities during the production process.

[0003] In related technologies, production line abnormality detection usually uses visual information or millimeter-wave radar information to detect, for example, whether workers have entered a specific area, or whether workers have a risk conflict with production equipment.

[0004] Regarding the above-mentioned related technologies, detection relies on visual information or millimeter-wave radar information. These two types of data are processed independently and are only linked through simple logic (such as threshold judgment). They are unable to deeply integrate features, resulting in a high missed detection rate of abnormal situations on the production line. There is still room for improvement. Summary of the invention

[0005] In order to improve the accuracy of production line abnormality detection and reduce the missed detection rate of production line abnormalities, the present application provides a production line abnormality detection method, system and terminal.

[0006] In the first aspect, the present application provides a method for detecting abnormal conditions in a production line, which adopts the following technical solution: A method for detecting abnormal conditions in a production line, comprising: Obtain the original image detection data and laser detection data of the production line; Controlling a preset visual modality processing branch to extract a visual feature vector according to the original image detection data; Controlling a preset laser mode processing branch to extract a laser feature vector according to the laser detection raw data; Input the visual feature vector and the laser feature vector into a preset cross-modal fusion module for feature fusion to generate a fusion feature matrix; Input the fused feature matrix into the preset classification model for classification to determine the type of production line abnormality; Prompt personnel based on the type of production line abnormality.

[0007] By adopting the above technical scheme, the visual modal processing branch is controlled to extract the visual feature vector according to the original image detection data, and the laser modal processing branch is controlled to extract the laser feature vector according to the original laser detection data, so that the visual feature vector and the laser feature vector are input into the trans-membrane fusion module for deep feature fusion to obtain the fusion feature matrix, and then the classification model is controlled to determine the type of production line abnormality according to the fusion feature matrix, thereby improving the accuracy of production line abnormality detection and reducing the missed detection rate of production line abnormalities.

[0008] Optionally, the steps of prompting personnel according to the type of production line abnormality include: Determine whether the abnormal type of the production line meets the requirements of the preset fire scene type; If it does not meet the requirements, the personnel will be prompted according to the preset stay away prompt information; If it meets the requirements, the personnel location and fire location are obtained; Assist personnel to leave the production line based on their location and fire location.

[0009] By adopting the above technical solution, when it is determined that the production line abnormality type does not meet the requirements of the fire scene type, personnel are prompted to stay away from the area based on the principle prompt information. When it meets the requirements of the fire scene type, personnel are assisted to leave the production line based on their location and the fire location, thereby improving the protection of personnel when production line abnormalities occur.

[0010] Optionally, the steps of assisting personnel to leave the production line according to the personnel location and the fire location include: Obtain safe evacuation paths based on personnel locations; Determine whether the fire location meets the requirements of a safe evacuation route; If it meets the requirements, the best evacuation path is obtained; If not, obtain the fire conditions of the safe evacuation route; Analyze the fire situation and the corresponding safe evacuation route to determine the best evacuation route; The preset auxiliary evacuation device is controlled according to the optimal evacuation path to assist personnel in leaving the production line.

[0011] By adopting the above technical solution, a safe evacuation path is determined according to the location of the personnel, and the fire location and fire situation are comprehensively considered in the safe evacuation path to select the best evacuation path, so that the auxiliary evacuation device is controlled by the best evacuation path to assist personnel in leaving the production line, thereby improving the safety of personnel evacuating the production line.

[0012] Optionally, the step of obtaining a safe evacuation path based on the personnel position includes: Determine the design safety path based on the relationship between the personnel position and the preset position path; Determine whether the personnel position meets the requirements of the preset low-level position; If it does not meet the requirements, the designed safe path is defined as the safe evacuation path; If it matches, get the window position; Establish additional safe routes based on window locations, occupant locations, and fire locations; The designed safety path and the additional safety path are associated to generate a safe evacuation path.

[0013] By adopting the above technical solution, the designed safety path is determined according to the relationship between the personnel position and the position path. When it is determined that the personnel position meets the requirements of the low-level position, an additional safety path is established according to the window position, the personnel position and the fire position, so as to obtain a safe evacuation path by associating the designed safety path and the additional safety path, so that personnel can leave the production line as quickly as possible, thereby improving the safety of personnel when evacuating from the production line.

[0014] Optionally, the step of controlling a preset auxiliary evacuation device to assist personnel in leaving the production line according to the optimal evacuation path includes: Determine the parameters of the starting device according to the relationship between the personnel position and the preset position device; Start the auxiliary evacuation device according to the parameters of the starting device, and control the auxiliary evacuation device to move to the personnel position; Control the auxiliary evacuation device according to the optimal evacuation path to guide personnel out of the production line and obtain the real-time evacuation location; Analyze real-time evacuation locations and optimal evacuation routes to determine projection paths; The auxiliary evacuation device is controlled according to the projection path to project the path to guide personnel away from the production line.

[0015] By adopting the above technical solution, the auxiliary evacuation device is started according to the parameters of the starting device, so that the auxiliary evacuation device can reach the personnel position as quickly as possible, thereby leading the personnel to leave the production line according to the optimal evacuation path, and projecting the projection path to guide the personnel, thereby improving the reliability of the auxiliary personnel evacuating the production line.

[0016] Optionally, the step of controlling the auxiliary evacuation device to guide personnel to leave the production line according to the optimal evacuation path includes: Control the auxiliary evacuation device to lead personnel away from the production line along the optimal evacuation path, and obtain the environmental parameters of the optimal evacuation path in real time; Determine whether the environmental parameters meet the requirements of the preset hazard parameters; If it does not meet the requirements, the environmental parameters of the optimal evacuation path will continue to be obtained in real time for cyclic judgment; If it meets the requirements, the auxiliary evacuation device is controlled to send a preset temperature reduction trigger signal; According to the temperature drop trigger signal, the preset fire-fighting device is controlled to spray water mist to achieve temperature drop and obtain the fire-fighting response situation; Control the auxiliary evacuation device to perform auxiliary cooling according to the fire response situation.

[0017] By adopting the above technical solution, when it is determined that the environmental parameters meet the requirements of the hazard parameters, the auxiliary evacuation device is controlled to send a cooling trigger signal, so that the fire-fighting device on the optimal evacuation path receives the cooling trigger signal and performs water mist spraying to achieve cooling, thereby reducing the impact of high temperature on personnel on the way to evacuate the production line.

[0018] Optionally, the step of controlling the auxiliary evacuation device to assist in cooling according to the fire response situation includes: Determine whether the fire response situation meets the requirements of the preset response situation; If it meets the requirements, the fire response situation will continue to be obtained for cyclic judgment; If it does not meet the requirements, the auxiliary evacuation device will be controlled to spray water mist above the personnel to achieve cooling.

[0019] By adopting the above technical solution, when it is determined that the fire response situation does not meet the requirements of the responded situation, the auxiliary evacuation device is controlled to spray water mist above the personnel, thereby ensuring that in areas where there are no fire-fighting devices or the fire-fighting devices are damaged, the auxiliary evacuation device can replace the fire-fighting device for cooling, thereby improving the safety of personnel during evacuation.

[0020] Optionally, the step of controlling the auxiliary evacuation device to spray water mist above the personnel to achieve cooling includes: Obtain information about auxiliary personnel; Determine whether the assistant status is a preset single-person assistant status or a preset multi-person assistant status; If it is a single-person auxiliary state, the auxiliary evacuation device is controlled to directly spray water mist to achieve cooling according to the preset single-person spray parameters; If it is in multi-person auxiliary state, obtain the multi-person center position and multi-person range; Analyze the multi-person range to determine the spray range; The auxiliary evacuation device is controlled to move to the multi-person center position to lead personnel to leave the production line, and the auxiliary evacuation device is controlled to spray water mist according to the spray range to achieve cooling.

[0021] By adopting the above technical solution, when it is determined that the auxiliary personnel is in a single-person auxiliary state, the auxiliary evacuation device is directly controlled to spray water mist according to the single-person spray parameters; when the auxiliary personnel is in a multi-person auxiliary state, the spray range is determined according to the multi-person range, so as to control the auxiliary evacuation device to move to the center position of the multi-person, and then spray water mist within the spray range to ensure that multiple people are cooled at the same time, thereby improving the safety of personnel during evacuation.

[0022] In the second aspect, the present application provides a production line abnormality detection system, which adopts the following technical solution: A production line abnormality detection system, comprising: An acquisition module is used to acquire image detection raw data and laser detection raw data; A memory, used to store a program of a production line abnormality detection method as described in any one of the above items; The program in the memory can be loaded and executed by the processor to implement a production line abnormality detection method as described in any one of the above items.

[0023] By adopting the above technical solution, the processor loads and executes a program of a production line abnormality detection method stored in the memory, thereby controlling the acquisition module to obtain a series of data related to the production line abnormality detection, controlling the visual modal processing branch to extract the visual feature vector according to the image detection raw data, and controlling the laser modal processing branch to extract the laser feature vector according to the laser detection raw data, thereby inputting the visual feature vector and the laser feature vector into the transmembrane state fusion module for deep feature fusion to obtain a fusion feature matrix, and then controlling the classification model to determine the production line abnormality type according to the fusion feature matrix, thereby improving the accuracy of production line abnormality detection and reducing the missed detection rate of production line abnormalities.

[0024] In a third aspect, the present application provides a smart terminal, which adopts the following technical solution: An intelligent terminal includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes a production line abnormality detection method as described in any one of the above items.

[0025] By adopting the above technical solution and operating the intelligent terminal, the processor is made to load and execute a computer program of a production line abnormality detection method stored in the memory, thereby controlling the visual modal processing branch to extract the visual feature vector according to the original image detection data, and controlling the laser modal processing branch to extract the laser feature vector according to the original laser detection data, thereby inputting the visual feature vector and the laser feature vector into the trans-membrane fusion module for deep feature fusion to obtain a fusion feature matrix, and then controlling the classification model to determine the production line abnormality type according to the fusion feature matrix, thereby improving the accuracy of production line abnormality detection and reducing the missed detection rate of production line abnormalities.

[0026] In summary, the present application includes at least one of the following beneficial technical effects: 1. By controlling the visual modal processing branch to extract the visual feature vector according to the original image detection data, and controlling the laser modal processing branch to extract the laser feature vector according to the original laser detection data, the visual feature vector and the laser feature vector are input into the transmembrane state fusion module for deep feature fusion to obtain the fusion feature matrix, and then the classification model is controlled to determine the type of production line abnormality according to the fusion feature matrix, thereby improving the accuracy of production line abnormality detection and reducing the missed detection rate of production line abnormalities; 2. By starting the auxiliary evacuation device according to the parameters of the starting device, the auxiliary evacuation device can reach the personnel position at the fastest speed, thereby leading the personnel to leave the production line according to the best evacuation path, and projecting the projection path to guide the personnel, thereby improving the reliability of the auxiliary personnel evacuating the production line; 3. When it is determined that the auxiliary personnel is in a single-person auxiliary state, the auxiliary evacuation device is directly controlled to spray water mist according to the single-person spray parameters; when the auxiliary personnel is in a multi-person auxiliary state, the spray range is determined according to the multi-person range, so as to control the auxiliary evacuation device to move to the center position of the multi-person, and then spray water mist within the spray range to ensure that multiple people are cooled at the same time, thereby improving the safety of personnel during evacuation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a method for detecting abnormal conditions in a production line in an embodiment of the present application.

[0028] Figure 2 It is a flowchart of the steps of prompting personnel according to the type of production line abnormality in an embodiment of the present application.

[0029] Figure 3 This is a flowchart of the steps of assisting personnel to leave the production line according to the personnel location and fire location in an embodiment of the present application.

[0030] Figure 4 This is a flowchart of the steps of obtaining a safe evacuation path based on personnel location in an embodiment of the present application.

[0031] Figure 5 It is a flow chart of the steps of controlling a preset auxiliary evacuation device to assist personnel in leaving the production line according to an optimal evacuation path in an embodiment of the present application.

[0032] Figure 6 It is a flow chart of the steps of controlling the auxiliary evacuation device to lead personnel out of the production line according to the optimal evacuation path in an embodiment of the present application.

[0033] Figure 7 It is a flow chart of the steps of controlling the auxiliary evacuation device to assist in cooling according to the fire response situation in an embodiment of the present application.

[0034] Figure 8It is a flow chart of the steps of controlling the auxiliary evacuation device to spray water mist above personnel to achieve cooling in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figures 1 to 8 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] The embodiment of the present application discloses a method for detecting production line abnormalities. After relevant sensors collect image detection raw data and laser detection raw data, the visual modal processing branch analyzes and processes the image detection raw data to extract visual feature vectors, and the laser modal processing branch analyzes and processes the laser detection raw data to extract laser feature vectors. Then, the cross-modal fusion module is controlled to fuse the visual feature vector and the laser feature vector to obtain a fused feature matrix, so that the classification model identifies the production line abnormality according to the fused feature matrix, obtains the production line abnormality type, and gives prompts, thereby improving the accuracy of production line abnormality detection and reducing the missed detection rate of production line abnormalities.

[0037] Reference Figure 1 The present application discloses a method for detecting abnormal conditions in a production line, comprising the following steps: Step S100: Acquire the original image detection data and laser detection data of the production line.

[0038] Among them, image detection raw data refers to the image data in the production line area, including two-dimensional images, video streams and depth maps, which can be captured by the camera installed in the production line and pre-processed by distortion correction, white balance and histogram equalization. Laser detection raw data refers to the data scanned by the laser in the production line area, including point cloud data. Each point contains coordinates, reflection intensity and timestamp, as well as Doppler velocity, such as target radial velocity, and vibration spectrum. Frequency domain features are extracted through FFT analysis and can be detected by solid-state LiDAR. The detection of image detection raw data and laser detection raw data requires time and space alignment through hardware synchronization and spatial calibration. GPS / PPS (pulse per second) signals or hardware trigger lines are used to synchronize the timestamps of the camera and LiDAR, and the spatial calibration is performed using the checkerboard calibration method.

[0039] Step S101: controlling a preset visual modality processing branch to extract a visual feature vector according to the original image detection data.

[0040] Among them, the visual modality processing branch refers to a model used to extract spatial information of people and objects in an image, and YOLOv7 is used in the embodiment of the present application.

[0041] The visual feature vector refers to the extracted target category, position coordinates and posture vector. Target detection locates the bounding boxes of people and objects through the Anchor-based mechanism. Posture estimation parses the human skeleton joints based on the key point detection network (such as the HRNet branch). Timing modeling uses optical flow or 3D convolution to track motion trajectories in the video stream.

[0042] Step S102: controlling a preset laser mode processing branch to extract a laser feature vector according to the laser detection raw data.

[0043] Among them, the laser modal processing branch refers to a model used to extract distance, speed and vibration signals from laser data. The LiDAR point cloud network is used in the embodiment of the present application.

[0044] The laser feature vector refers to the extracted target distance, speed and vibration spectrum. Point cloud preprocessing is first performed, and VoxelNet or PointPillars is used to voxelize the point cloud. Then, the spatial distribution characteristics are extracted through the 3D convolution kernel. The radial velocity of the moving target is separated through FFT spectrum analysis, and the nanoscale vibration frequency of the device surface is extracted by wavelet transform.

[0045] Step S103: inputting the visual feature vector and the laser feature vector into a preset cross-modal fusion module for feature fusion to generate a fusion feature matrix.

[0046] Among them, the cross-modal fusion module refers to the module that performs feature-level fusion, including spatiotemporal alignment, attention interaction, and feature enhancement.

[0047] The fused feature matrix refers to the feature matrix obtained after feature-level fusion of the visual feature vector and the laser feature vector. First, based on the hardware synchronization signal and the calibrated external parameter matrix, the two-modal features are mapped to the spatiotemporal coordinates. Then, a cross-modal Transformer encoder is used to calculate the attention weights with visual features as queries and laser features as keys / values ​​to generate a fused feature matrix. Finally, the spatiotemporal correlation of the fused features is modeled through a 4-layer multi-head self-attention layer, and position encoding is added to enhance spatial sensitivity. For example, when a person is detected crossing the boundary, the visual posture features will be automatically associated with the instantaneous speed features of the LiDAR.

[0048] Step S104: input the fused feature matrix into a preset classification model for classification to determine the abnormal type of the production line.

[0049] Among them, the classification model refers to a model that classifies production line abnormalities based on a fused feature matrix. In the embodiment of the present application, a multi-task BERT variant model is adopted to jointly train classification and regression tasks through an uncertainty weighted loss function.

[0050] The production line abnormality type refers to the abnormality type that occurs in the production line. The fused feature matrix is ​​input into the classification model, and the abnormality classification is performed simultaneously: the category probability of personnel intrusion, equipment vibration exceeding the limit, and other extreme abnormal situations is output through the fully connected layer and the Softmax function; vibration regression: the L1 loss function is used to predict the micron-level value of the vibration amplitude; positioning correction: the error of the target position is compensated based on the Kalman filter prior, and the calibrated three-dimensional coordinates are output.

[0051] Step S105: Prompt personnel according to the type of production line abnormality.

[0052] Among them, after obtaining the abnormal type of the production line, voice broadcasts are made to personnel according to the abnormal type of the production line, or auxiliary personnel are kept away from the isolation area or production line. For specific methods, refer to Figure 2 steps to ensure the safety of personnel when abnormal situations occur on the production line.

[0053] Reference Figure 2 The steps for prompting personnel based on the type of production line abnormality include: Step S200: Determine whether the production line abnormality type meets the requirements of the preset fire scene type.

[0054] Among them, the fire scene type refers to the abnormal situation of the production line as the occurrence of a fire scene, and the fire scene type requirement refers to consistency with the fire scene type.

[0055] The processing terminal determines whether the production line abnormality type is consistent with the fire scene type, thereby determining whether a fire has occurred in the production line.

[0056] Step S201: If not, the person is prompted according to the preset stay away prompt information.

[0057] Among them, if the processing terminal determines that the production line abnormality type is inconsistent with the fire scene type, it indicates that the production line abnormality type is that personnel enter the isolation area or the equipment vibration exceeds the limit. Therefore, personnel can be prompted to stay away from the isolation area or equipment according to the stay away prompt information.

[0058] The stay-away prompt information refers to a voice message that prompts personnel to stay away from the equipment and the isolated area, and is stored in the processing terminal by the operator.

[0059] Step S202: If it is in compliance, the personnel location and the fire location are obtained.

[0060] Among them, if the processing terminal determines that the production line abnormality type is consistent with the fire scene type, it means that a fire has occurred on the production line. At this time, personnel need to leave the production line. Therefore, the personnel position and the fire position are detected to provide data support for subsequent auxiliary personnel to leave the production line.

[0061] The personnel position refers to the position of the personnel in the production line, and the fire position refers to the position where the fire occurs in the production line. The visual modal processing branch extracts the target position coordinates based on the target features of the personnel and the fire.

[0062] Step S203: assisting personnel to leave the production line according to their positions and the fire position.

[0063] Among them, after determining the personnel location and fire location, assist personnel to leave the production line according to the personnel location and fire location. The specific method is as follows Figure 3 steps to ensure the safety of personnel when abnormalities occur on the production line.

[0064] Reference Figure 3 , the steps to assist personnel to leave the production line according to the personnel location and fire location include: Step S300: Obtain a safe evacuation path based on the personnel's position.

[0065] The safe evacuation path refers to the safe path for personnel to evacuate from the production line under normal circumstances. For specific methods of obtaining it, refer to Figure 3 steps.

[0066] Step S301: Determine whether the fire location meets the requirements of a safe evacuation route.

[0067] The requirement of the safe evacuation path is that the fire location is outside the safe evacuation path. The processing terminal determines whether the fire location is outside the safe evacuation path, thereby determining whether the safe evacuation path can be used.

[0068] Step S3011: If it meets the requirements, the best evacuation path is obtained.

[0069] Among them, if the processing terminal determines that the fire location is outside the safe evacuation path, it means that the safe evacuation path can be used, so the best evacuation path is called to provide data support for subsequent auxiliary personnel to safely stay away from the production line.

[0070] The best evacuation path refers to the safe evacuation path without fire and the shortest path. The processing terminal selects the path without fire in the safe evacuation path, and then compares the length of such paths to select the shortest path as the best evacuation path.

[0071] Step S3012: If not, obtain the fire situation of the safe evacuation route.

[0072] Among them, if the processing terminal determines that the fire location is within the safe evacuation path, it means that a fire has occurred within the safe evacuation path. Therefore, the fire situation is detected to provide data support for the subsequent determination of the best evacuation path.

[0073] The fire condition refers to the size of the fire, which is determined by extracting the area of ​​the fire by the visual modality processing branch.

[0074] Step S302: Analyze the fire situation and the corresponding safe evacuation path to determine the best evacuation path.

[0075] Among them, the best evacuation path in this step is consistent with the best evacuation path in step S3011. The processing terminal compares the fire conditions in the safe evacuation path, and selects the path with the smallest fire condition and the shortest path as the best evacuation path.

[0076] Step S303: controlling the preset auxiliary evacuation device according to the optimal evacuation path to assist personnel to leave the production line.

[0077] Among them, after determining the best evacuation path, the processing terminal controls the auxiliary evacuation device to assist personnel to leave the production line. The specific method is as follows Figure 5 steps to ensure the safety of personnel evacuation.

[0078] An auxiliary evacuation device refers to a device that assists personnel in leaving the production line. A drone can be used to guide personnel away from the production line. The drone carries a spray device to achieve cooling, and also carries a projection device to project the path onto the ground to guide personnel.

[0079] Reference Figure 4 ,The steps of obtaining a safe evacuation path based on the personnel location correspondence include: Step S400: Determine a designed safe path according to the personnel position and a preset position path relationship.

[0080] The position-path relationship refers to the correspondence between different positions and safety paths. The operator forms a mapping table by mapping different safety paths to the positions on the paths one by one.

[0081] The designed safety path refers to the safety path designed in the production line, which is found by the processing terminal in the mapping table corresponding to the position-path relationship according to the personnel position.

[0082] Step S401: Determine whether the personnel position meets the preset low-level position requirements.

[0083] Among them, the low-level position means that the position of the personnel is on the first floor, and the requirements of the low-level position refer to being consistent with the low-level position.

[0084] The processing terminal determines whether the personnel position is consistent with the lower level position, thereby determining whether the personnel is on the first level.

[0085] Step S4011: If not, define the designed safety path as a safe evacuation path.

[0086] Among them, if the processing terminal determines that the position of the person is inconsistent with the lower-level position, it means that the person is at a high altitude and the window cannot be used as an evacuation route. Therefore, the designed safe path is defined as the safe evacuation path.

[0087] Step S4012: If it meets the requirements, obtain the window position.

[0088] Among them, if the processing terminal determines that the person's position is consistent with the position on the lower floor, it means that the person is on the first floor and the window can be used as an evacuation channel. Therefore, the window position is called to provide data support for the subsequent determination of a safe evacuation path.

[0089] The window position refers to the position of the windows in the production line area. The window positions in the production line area are uploaded by the operator and stored in the processing terminal.

[0090] Step S402: Establish an additional safety path according to the window position, personnel position and fire position.

[0091] Among them, the additional safety path refers to the path established with windows as the evacuation channel. The processing terminal establishes the evacuation path according to the window position and the personnel position, and then selects the path in the evacuation path that does not overlap with the fire position as the additional safety path.

[0092] Step S403: Associating the designed safety path and the additional safety path to generate a safe evacuation path.

[0093] The safe evacuation path in this step is consistent with the safe evacuation path in step S300, and is formed by the processing terminal storing the designed safe path and the additional safe path in the same data packet.

[0094] Reference Figure 5 The steps of controlling the preset auxiliary evacuation device to assist personnel to leave the production line according to the optimal evacuation path include: Step S500: Determine the parameters of the activation device according to the personnel position and the preset position-device relationship.

[0095] Among them, the position-device relationship refers to the correspondence between the personnel position and the nearest auxiliary evacuation device. The operator forms a mapping table by matching the position where the auxiliary evacuation device is set with the position that the device can assist, so that each auxiliary evacuation device assists all positions in an area.

[0096] The activation device parameters refer to the auxiliary evacuation devices that need to be activated, which are found by the processing terminal in the mapping table corresponding to the position-device relationship according to the personnel position.

[0097] Step S501: starting the auxiliary evacuation device according to the starting device parameters, and controlling the auxiliary evacuation device to move to the personnel position.

[0098] Among them, after determining the parameters of the starting device, the processing terminal starts the corresponding auxiliary evacuation device according to the parameters of the starting device, and controls the auxiliary evacuation device to move to the personnel position, so as to prepare for the subsequent auxiliary personnel to evacuate the production line.

[0099] Step S502: Control the auxiliary evacuation device according to the optimal evacuation path to lead the personnel to leave the production line, and obtain the real-time evacuation position.

[0100] Among them, after the auxiliary evacuation device reaches the personnel position, the auxiliary evacuation device guides the personnel to leave the production line along the best evacuation path. The specific method is as follows Figure 6 steps and detect the real-time evacuation location to provide data support for subsequent guidance.

[0101] The real-time evacuation position refers to the real-time position of the auxiliary evacuation device during the evacuation process, and the position coordinates can be obtained by extracting the image of the auxiliary evacuation device by the visual modality processing branch.

[0102] Step S503: Analyze the real-time evacuation position and the best evacuation path to determine the projection path.

[0103] The projected path refers to a partial path projected onto the ground, which is obtained by the processing terminal identifying a portion of the evacuation path below the real-time evacuation position based on the position of the real-time evacuation position on the optimal evacuation path.

[0104] Step S504: Control the auxiliary evacuation device to project a path according to the projected path to guide personnel to leave the production line.

[0105] Among them, after determining the projection path, the projection device in the auxiliary evacuation device projects the projection path on the ground, so that personnel can see the exit path more clearly when leaving the production line, ensuring the accuracy and speed of evacuation.

[0106] Reference Figure 6 The steps of controlling the auxiliary evacuation device to lead personnel out of the production line according to the optimal evacuation path include: Step S600: Control the auxiliary evacuation device to lead personnel to leave the production line along the optimal evacuation path, and obtain the environmental parameters of the optimal evacuation path in real time.

[0107] Among them, after the auxiliary evacuation device reaches the personnel's location, the processing terminal controls the auxiliary evacuation device to lead the personnel away from the production line along the optimal evacuation path, and detects the environmental parameters of the optimal evacuation path to provide data support for subsequent determination of whether fire intervention is needed.

[0108] Environmental parameters refer to the environment-related parameters on the optimal evacuation path, including temperature and flame conditions. The temperature is detected by the temperature sensor on the auxiliary evacuation device, and the flame condition is obtained by extracting features from the images collected by the auxiliary evacuation device by the visual modal processing branch.

[0109] Step S601: Determine whether the environmental parameters meet the requirements of the preset danger parameters.

[0110] Among them, the danger parameter refers to the environmental parameter of excessive temperature or flame on the evacuation path, and the requirement of the danger parameter refers to the consistency with the situation corresponding to the danger parameter.

[0111] The processing terminal determines whether the situation corresponding to the environmental parameters is consistent with the situation corresponding to the danger parameters, thereby determining whether fire intervention is required on the best evacuation path.

[0112] Step S6011: If not, continue to obtain the environmental parameters of the best evacuation path in real time for cyclic judgment.

[0113] Among them, if the processing terminal determines that the situation corresponding to the environmental parameters is inconsistent with the situation corresponding to the danger parameters, it means that the temperature on the best evacuation path is low and there is no flame, so no fire intervention is required, and the environmental parameters of the best evacuation path continue to be detected to continue to pay attention to the environmental changes of the best evacuation path.

[0114] Step S6012: If the conditions are met, the auxiliary evacuation device is controlled to send a preset temperature reduction trigger signal.

[0115] Among them, if the processing terminal determines that the situation corresponding to the environmental parameters is consistent with the situation corresponding to the danger parameters, it indicates that the temperature is high or flames appear on the best evacuation path, so firefighting intervention is required, so that the auxiliary evacuation device sends a cooling trigger signal to control the fire-fighting device to spray water mist.

[0116] The cooling trigger signal refers to the signal that controls the fire-fighting device to spray water mist. It is sent by the auxiliary evacuation device and received and responded by the fire-fighting device.

[0117] Step S602: Control the preset fire-fighting device to spray water mist to achieve cooling according to the temperature reduction trigger signal, and obtain the fire-fighting response status.

[0118] Among them, the fire-fighting device receives and responds to the cooling trigger signal, thereby spraying water mist to achieve cooling on the optimal evacuation path, ensure the safety of personnel evacuation, and detect the fire-fighting response situation to provide data support for the subsequent determination of whether auxiliary evacuation device auxiliary spraying is needed.

[0119] Firefighting devices refer to fire sprinklers installed in the production line for fire extinguishing. Firefighting response refers to the response of firefighting devices, including response and non-response. Firefighting devices send feedback signals to the processing terminal. If the processing terminal receives the feedback signal, it is determined to have responded. If it does not receive the feedback signal, it is determined to have not responded.

[0120] Step S603: Control the auxiliary evacuation device to perform auxiliary cooling according to the fire response situation.

[0121] Among them, when receiving the fire response situation, the processing terminal controls the auxiliary evacuation device to perform auxiliary cooling according to the fire response situation. The specific method is referred to Figure 7 steps to ensure that when the fire-fighting device does not respond, the auxiliary evacuation device can promptly replace the fire-fighting device for cooling.

[0122] Reference Figure 7 The steps of controlling the auxiliary evacuation device to assist in cooling according to the fire response situation include: Step S700: Determine whether the fire response situation meets the requirements of the preset response situation.

[0123] Among them, the responded situation refers to the fire-fighting device spraying water mist normally in response to the fire trigger signal, and the requirements of the responded situation refer to being consistent with the responded situation.

[0124] The processing terminal determines whether the fire response situation is consistent with the responded situation, thereby determining whether the fire-fighting equipment is working properly.

[0125] Step S701: If it meets the requirements, continue to obtain the fire response situation and make a cyclic judgment.

[0126] Among them, if the processing terminal determines that the fire response situation is consistent with the responded situation, it means that the fire-fighting device is working normally, so the fire response situation continues to be detected, thereby continuing to pay attention to the working condition of the fire-fighting device.

[0127] Step S702: If not met, the auxiliary evacuation device is controlled to spray water mist above the personnel to achieve cooling.

[0128] If the processing terminal determines that the fire response situation is inconsistent with the responded situation, it means that the fire-fighting device cannot spray water mist. Therefore, the auxiliary evacuation device is controlled to spray water mist above the personnel, thereby replacing the fire-fighting device for cooling. For specific methods, refer to Figure 8 steps.

[0129] Reference Figure 8 The steps of controlling the auxiliary evacuation device to spray water mist above the personnel to achieve cooling include: Step S800: Obtain information about auxiliary personnel.

[0130] The auxiliary personnel situation refers to the number of personnel evacuated by the auxiliary evacuation device, which is obtained by extracting features from images collected by the auxiliary evacuation device by the visual modal processing branch.

[0131] Step S801: Determine whether the assistant status is a preset single-person assistant status or a preset multi-person assistant status.

[0132] Among them, the single-person assistance state means that the auxiliary evacuation device only assists one person to evacuate, and the multi-person assistance state means that the auxiliary evacuation device assists multiple people to evacuate.

[0133] The processing terminal determines whether the assistant is in a single-person assistant state or a multi-person assistant state, thereby determining how to spray water mist.

[0134] Step S8011: If it is a single-person auxiliary state, the auxiliary evacuation device is controlled to directly spray water mist to achieve cooling according to the preset single-person spray parameters.

[0135] Among them, if the processing terminal determines that the auxiliary personnel situation is a single-person auxiliary state, it means that the auxiliary evacuation device only assists one person to evacuate. Therefore, the auxiliary evacuation device is controlled according to the single-person spray parameter to spray water mist above the person to achieve cooling.

[0136] The single-person spray parameter refers to the range of the auxiliary evacuation device when spraying water mist on a single person. The specific value is determined by the operator based on actual conditions.

[0137] Step S8012: If it is a multi-person assistance state, obtain the multi-person center position and multi-person range.

[0138] Among them, if the processing terminal determines that the auxiliary personnel situation is a multi-person auxiliary state, it means that the auxiliary evacuation device assists multiple people to evacuate at the same time. Therefore, the central position and range of multiple people are detected to provide data support for the subsequent control of the auxiliary evacuation device to spray water mist.

[0139] The multi-person center position refers to the middle position of multiple people, and the multi-person range refers to the range radius of the personnel, which is obtained by extracting features from the images collected by the auxiliary evacuation device by the visual modal processing branch.

[0140] Step S802: Analyze the range of multiple people to determine the spraying range.

[0141] Among them, the spray range refers to the radius of the water mist sprayed by the auxiliary evacuation device, that is, the radius corresponding to the range of multiple people.

[0142] Step S803: Control the auxiliary evacuation device to move to the multi-person center position to lead the personnel to leave the production line, and control the auxiliary evacuation device to spray water mist according to the spraying range to achieve cooling.

[0143] Among them, after determining the multi-person center position, the processing terminal controls the auxiliary evacuation device to move to the multi-person center position to lead personnel away from the production line, and controls the auxiliary evacuation device to spray water mist within the spray range at the multi-person center position, thereby effectively cooling down all people.

[0144] Based on the same inventive concept, the embodiment of the present application provides a production line abnormality detection system, including: An acquisition module is used to acquire image detection raw data, laser detection raw data, personnel location, fire location, safe evacuation path, optimal evacuation path, fire situation, window location, real-time evacuation location, environmental parameters, fire response situation, auxiliary personnel situation, multi-person center location and multi-person range; A memory, used to store a program of a production line abnormality detection method; The program in the memory can be loaded and executed by the processor to implement a method for detecting abnormal conditions in a production line.

[0145] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0146] An embodiment of the present application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed by a method for detecting abnormal conditions in a production line.

[0147] Computer storage media include, for example, USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks, and other media that can store program codes.

[0148] Based on the same inventive concept, an embodiment of the present application provides an intelligent terminal, including a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes a method for detecting abnormal conditions in a production line.

[0149] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0150] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A method for detecting abnormal conditions in a production line, characterized in that: include: Obtain the original image detection data and laser detection data of the production line; Controlling a preset visual modality processing branch to extract a visual feature vector according to the original image detection data; Controlling a preset laser mode processing branch to extract a laser feature vector according to the laser detection raw data; Input the visual feature vector and the laser feature vector into a preset cross-modal fusion module for feature fusion to generate a fusion feature matrix; Input the fused feature matrix into the preset classification model for classification to determine the type of production line abnormality; Prompt personnel based on the type of production line abnormality.

2. A production line abnormality detection method according to claim 1, characterized in that: The steps for prompting personnel based on the type of production line abnormality include: Determine whether the abnormal type of the production line meets the requirements of the preset fire scene type; If it does not meet the requirements, the personnel will be prompted according to the preset stay away prompt information; If it meets the requirements, the personnel location and fire location are obtained; Assist personnel to leave the production line based on their location and fire location.

3. A production line abnormality detection method according to claim 2, characterized in that: The steps to assist personnel to leave the production line based on their location and fire location include: Obtain safe evacuation paths based on personnel locations; Determine whether the fire location meets the requirements of a safe evacuation route; If it meets the requirements, the best evacuation path is obtained; If not, obtain the fire conditions of the safe evacuation route; Analyze the fire situation and the corresponding safe evacuation route to determine the best evacuation route; The preset auxiliary evacuation device is controlled according to the optimal evacuation path to assist personnel in leaving the production line.

4. A production line abnormality detection method according to claim 3, characterized in that: The steps of obtaining a safe evacuation path based on personnel location correspondence include: Determine the design safety path based on the relationship between the personnel position and the preset position path; Determine whether the personnel position meets the requirements of the preset low-level position; If it does not meet the requirements, the designed safe path is defined as the safe evacuation path; If it matches, get the window position; Establish additional safe routes based on window locations, occupant locations, and fire locations; The designed safety path and the additional safety path are associated to generate a safe evacuation path.

5. A production line abnormality detection method according to claim 3, characterized in that: The steps of controlling the preset auxiliary evacuation device according to the optimal evacuation path to assist personnel to leave the production line include: Determine the parameters of the starting device according to the relationship between the personnel position and the preset position device; Start the auxiliary evacuation device according to the parameters of the starting device, and control the auxiliary evacuation device to move to the personnel position; Control the auxiliary evacuation device according to the optimal evacuation path to guide personnel out of the production line and obtain the real-time evacuation location; Analyze real-time evacuation locations and optimal evacuation routes to determine projection paths; The auxiliary evacuation device is controlled according to the projection path to project the path to guide personnel away from the production line.

6. A production line abnormality detection method according to claim 5, characterized in that: The steps of controlling the auxiliary evacuation device to guide personnel out of the production line according to the optimal evacuation path include: Control the auxiliary evacuation device to lead personnel away from the production line along the optimal evacuation path, and obtain the environmental parameters of the optimal evacuation path in real time; Determine whether the environmental parameters meet the requirements of the preset hazard parameters; If it does not meet the requirements, the environmental parameters of the optimal evacuation path will continue to be obtained in real time for cyclic judgment; If it meets the requirements, the auxiliary evacuation device is controlled to send a preset temperature reduction trigger signal; According to the temperature drop trigger signal, the preset fire-fighting device is controlled to spray water mist to achieve temperature drop and obtain the fire-fighting response situation; Control the auxiliary evacuation device to perform auxiliary cooling according to the fire response situation.

7. A production line abnormality detection method according to claim 6, characterized in that: The steps for controlling the auxiliary evacuation device to assist in cooling according to the fire response situation include: Determine whether the fire response situation meets the requirements of the preset response situation; If it meets the requirements, the fire response situation will continue to be obtained for cyclic judgment; If it does not meet the requirements, the auxiliary evacuation device will be controlled to spray water mist above the personnel to achieve cooling.

8. A production line abnormality detection method according to claim 7, characterized in that: The steps of controlling the auxiliary evacuation device to spray water mist above the personnel to achieve cooling include: Obtain information about auxiliary personnel; Determine whether the assistant status is a preset single-person assistant status or a preset multi-person assistant status; If it is a single-person auxiliary state, the auxiliary evacuation device is controlled to directly spray water mist to achieve cooling according to the preset single-person spray parameters; If it is in multi-person auxiliary state, obtain the multi-person center position and multi-person range; Analyze the multi-person range to determine the spray range; The auxiliary evacuation device is controlled to move to the multi-person center position to lead personnel to leave the production line, and the auxiliary evacuation device is controlled to spray water mist according to the spray range to achieve cooling.

9. A production line abnormality detection system, characterized in that: include: An acquisition module is used to acquire image detection raw data and laser detection raw data; A memory, used to store a program of a production line abnormality detection method according to any one of claims 1 to 8; The program in the memory can be loaded and executed by the processor to implement a production line abnormality detection method as described in any one of claims 1 to 8.

10. An intelligent terminal, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes a method for detecting abnormal conditions in a production line as described in any one of claims 1 to 8.

Images