Safety channel blockage detection method, system and equipment and storage medium

Abstract

The invention discloses a security channel blockage detection method, system and device and a storage medium, the method is based on a security channel blockage detection system, the system comprises an intelligent terminal and a cloud server, and the intelligent terminal is used for obtaining a to-be-detected image; judging whether the monitoring video is changed or not; when the monitoring video changes, judging whether the change is caused by pedestrians or not; when the judgment result is no, judging that the to-be-detected image is in a suspected alarm state; when N continuous to-be-detected images are judged to be in the suspected alarm state, alarm images are generated, and alarm information carrying the alarm images is sent to a cloud server; the cloud server is used for calling a preset multi-modal large model to perform secondary judgment on the alarm image after receiving the alarm information; and when it is judged that there is an obstacle in the alarm image, alarm information is generated and sent to a preset user terminal. According to the scheme, the blockage of the safety channel can be efficiently and accurately detected.

Description

Technical Field

[0001] The present application relates to the technical field of factory safety management and control, and specifically to a safety passage blockage detection method, system, device and storage medium. Background Art

[0002] Recently, factories usually have safety passages for employees to evacuate and escape quickly when an accident occurs. However, due to various reasons such as negligence of personnel and non-compliance with regulations, various debris often appear in the safety passages, causing the safety passages to be blocked. When an accident really occurs, it causes a safety accident. It can be seen that the unobstructed safety passages play an important role in crowd evacuation and accident escape. Therefore, it is particularly important to promptly report when the safety passages are blocked and clear the blockages in time.

[0003] There have been many attempts to detect the blockage of safety passages. Initially, the solution mainly used manual viewing of surveillance videos and real-time monitoring. This solution requires monitoring personnel to maintain concentration for a long time and with high intensity. Factories cannot deploy this solution on a large scale. It has disadvantages such as time-consuming and labor-intensive consumption of manpower costs. Therefore, this method has been basically cancelled. Later, a solution using the similarity of image pixels was used to detect the blockage of safety passages. However, this solution has a large difference in pixel values ​​between day and night in the safety passage, and it is easy to trigger the sensor lights in the safety passage when pedestrians need to pass through the safety passage for daily production activities, resulting in many false alarms. Therefore, this method has also been basically cancelled.

[0004] With the development of computer vision technology, a solution for detecting blockages using target detection technology has emerged. This solution detects whether there are objects in the channel to achieve blockage detection. However, due to the complex factory environment, the use of traditional target detection solutions for safe channel blockage detection will not be able to cover all aspects. For example, pedestrians often make misjudgments when carrying production materials. In addition, there are many types of blocking objects, and there are no fixed category restrictions, making data acquisition difficult, resulting in the inability to exhaustively enumerate blockages in the target detection task, resulting in insufficient algorithm performance.

[0005] Therefore, how to provide a safety channel blockage detection method suitable for a factory environment, which can accurately detect channel blockage problems in different environments in the factory, is a technical problem that urgently needs to be solved by technical personnel in this field. Summary of the invention

[0006] In order to solve the above problems, the present application provides a safety channel blockage detection method, system, device and storage medium.

[0007] In a first aspect, an embodiment of the present application provides a safety channel blockage detection method, the method is based on a safety channel blockage detection system, the safety channel blockage detection system includes a smart terminal and a cloud server, the smart terminal is used to perform the following processing:

[0008] Acquire an image to be detected, where the image to be detected is the latest video frame read from the surveillance video;

[0009] Based on the image to be detected and a preset evacuation status library, determining whether the surveillance video has changed;

[0010] When the surveillance video changes, determining whether the change is caused by a pedestrian;

[0011] When the judgment result is no, determining that the image to be detected is in a suspected alarm state;

[0012] When N consecutive images to be detected are determined to be in a suspected alarm state, a detection area is marked in the Nth image to be detected to obtain an alarm image, where N is a preset alarm quantity threshold;

[0013] Sending the warning information carrying the warning image to the cloud server;

[0014] The cloud server is used to perform the following processing:

[0015] After receiving the warning information, calling a preset multi-modal large model to perform a secondary judgment on the warning image;

[0016] When it is determined that there is an obstacle in the warning image, warning information is generated and sent to a preset user terminal.

[0017] In some embodiments, judging whether the surveillance video has changed based on the image to be detected and a preset evacuation status library includes:

[0018] According to a preset area to be detected, the image to be detected is cropped to obtain a partial image to be detected;

[0019] sequentially identifying the similarity between the partial image to be detected and each evacuation state image in a preset evacuation state image library, and taking the maximum similarity as the determination similarity;

[0020] If the determined similarity is less than a preset similarity threshold, it is determined that the surveillance video has changed.

[0021] In some embodiments, the step of sequentially identifying the similarity between the partial image to be detected and each evacuation state diagram in a preset evacuation state diagram library includes:

[0022] Using a preset FAST feature point matching algorithm to perform feature point detection and matching on the local image to be detected and the target evacuation state diagram, the target evacuation state diagram being any one of all the evacuation state diagrams;

[0023] The similarity between the local image to be detected and the target evacuation state map = the number of matched feature points*2 / the total number of detected feature points.

[0024] In some embodiments, when the determination result is that the surveillance video has changed, determining whether the change is caused by a pedestrian includes:

[0025] When the judgment result is that the surveillance video has changed, the preset YOLOv5 target detection algorithm is used to identify whether there are pedestrian features in the image to be detected;

[0026] If the pedestrian feature does not exist, the judgment result is set to no;

[0027] If the recognition result shows that the pedestrian feature exists, the boundary box of the pedestrian feature is compared with a preset human feature area;

[0028] When the boundary box of the pedestrian feature and the preset human feature area have an intersection, the judgment result is set to yes;

[0029] When there is no intersection between the boundary box of the pedestrian feature and the preset human feature area, the judgment result is set to no.

[0030] In some embodiments, before acquiring the image to be detected, the intelligent terminal further performs the following processing:

[0031] Obtaining a standard video frame selected by a user from a surveillance video, wherein the standard video frame is a picture of a safe passage in an unobstructed state in the surveillance video;

[0032] Based on the area to be detected selected by the user, each of the standard video frames is cropped to obtain an evacuation state diagram, and the evacuation state diagram library is generated;

[0033] Taking the center point of the area to be detected as the center, the area to be detected is enlarged by a specified multiple to obtain a human body feature area.

[0034] In some embodiments, for the cloud server, after receiving the alarm information, calling a preset multimodal large model to perform a secondary judgment on the alarm image includes:

[0035] After receiving the alarm information, calling the preset InternVL multimodal large model;

[0036] Using the warning image in the warning information as the image picture input of the InternVL multimodal large model, and calling the preset text information as the prompt text input of the InternVL multimodal large model;

[0037] The image picture input and prompt text input are processed respectively through the InternVL multimodal large model to obtain a secondary judgment result for the warning image.

[0038] In some embodiments, in the step of processing the image input and the prompt text input respectively through the InternVL multimodal large model to obtain a secondary judgment result for the warning image, the cloud server calls the InternVL multimodal large model to perform the following processing:

[0039] Performing a text processing flow on the prompt text input to obtain an output of the text processing flow, i.e., a text vector;

[0040] Performing an image processing process on the image input, i.e., the alarm image, to obtain an output of the image processing process, i.e., an image feature vector;

[0041] Concatenate the text vector and the image feature vector to obtain a final feature vector;

[0042] The final feature vector is input into a preset InternL M2 Chat 20B language model to obtain a secondary judgment result for the warning image.

[0043] In a second aspect, an embodiment of the present application provides a safety channel blockage detection system, including a smart terminal and a cloud server, wherein the smart terminal includes:

[0044] An image acquisition module is used to acquire an image to be detected, where the image to be detected is the latest video frame read from the surveillance video;

[0045] An image change determination module, used to determine whether the monitoring video has changed based on the image to be detected and a preset evacuation status library;

[0046] A pedestrian feature determination module, used to determine whether the change is caused by a pedestrian when the monitoring video changes;

[0047] The pedestrian feature determination module is further configured to determine that the image to be detected is in a suspected alarm state when the determination result is negative;

[0048] An alarm image generation module is used to mark the area to be detected in the Nth image to be detected to obtain an alarm image when N consecutive images to be detected are determined to be in a suspected alarm state, where N is a preset alarm quantity threshold;

[0049] An alarm uploading module is used to send the alarm information carrying the alarm image to a preset cloud server;

[0050] The cloud server includes:

[0051] A determination module, configured to call a preset multimodal large model to perform a secondary determination on the warning image after receiving the warning information;

[0052] The warning module generates and sends warning information to a preset user terminal when it is determined that there is an obstacle in the warning image.

[0053] In a third aspect, an embodiment of the present application provides an intelligent device, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the safety channel blockage detection method as described in the first aspect is implemented.

[0054] In a fourth aspect, an embodiment of the present application provides a storage medium having a program stored thereon, and when the program is executed by a processor, it is used to implement the safety channel blockage detection method as described in the first aspect.

[0055] The technical solution of this application has at least the following advantages:

[0056] 1. Use the smart terminal to detect the latest video frame in the surveillance video, that is, the image to be detected, to determine whether the surveillance video has changed, that is, whether there is a risk of blockage. When it is determined that there is a risk of blockage, the smart terminal is used to further determine whether the change is caused by pedestrians, so as to eliminate the interference of pedestrians on the detection results and improve the accuracy. After eliminating the interference of pedestrian factors, the smart terminal will determine that the image to be detected is in a suspected alarm state, that is, a suspected blockage has occurred. When the suspected blockage lasts for a sufficient period of time, the control terminal will further send an alarm message to the cloud server, and the cloud server will then make a final secondary judgment to finally confirm whether the safety channel is blocked. Through multiple judgments, the detection of blockage in the safety channel can be accurately achieved;

[0057] 2. In the process of determining whether the image has changed, the detection is performed on the designated area rather than the debris itself. Therefore, there is no need to enumerate the types of blockages, which reduces the training cost of the algorithm and avoids the possibility of insufficient algorithm performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 is a flow chart of a safety channel blockage detection method provided by an exemplary embodiment of the present application;

[0060] Figure 2 It is a flowchart for reflecting the evacuation state library and human feature area configuration process in the safety passage blockage detection method provided by an exemplary embodiment of the present application;

[0061] Figure 3.1 and Figure 3.2 They are respectively schematic diagrams of a standard video frame provided by an exemplary embodiment of the present application;

[0062] Figure 4 is a schematic diagram of an evacuation state diagram provided by an exemplary embodiment of the present application;

[0063] Figure 5 is a schematic diagram of an image to be detected provided by an exemplary embodiment of the present application, which is used to reflect a boundary box of an area to be detected marked with white lines;

[0064] Figure 6 is a flow chart of S20 in a safety passage blockage detection method provided by an exemplary embodiment of the present application;

[0065] Figure 7 is a system block diagram of a safety passage blockage detection system provided by an exemplary embodiment of the present application;

[0066] Figure 8 It is a schematic diagram of the structure of a smart device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0069] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0070] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0071] The present application provides a safety passage blockage detection method, which is based on a safety passage blockage detection system and is used to monitor safety passages in a factory. The safety passage blockage detection system includes an intelligent terminal and a cloud server, wherein the intelligent terminal may be a computer, and is connected to a signal of a monitoring device disposed in the safety passage, so as to receive and process monitoring videos uploaded by the monitoring device. The monitoring device may be a camera, and is fixedly installed in safety passages such as stairs and safety doors. For the same monitoring device, its installation angle and installation position are fixed, so as to ensure the consistency of the viewing angle of the acquired monitoring video. Reference Figure 1 , the safety channel blockage detection method includes the following steps:

[0072] S10: The smart terminal obtains the image to be detected.

[0073] The image to be detected is the latest video frame read from the surveillance video.

[0074] For example, the monitoring device collects monitoring videos of the safety channel in real time and transmits them to the smart terminal. For any monitoring video, the smart terminal will identify the latest video frame in real time.

[0075] S20: The intelligent terminal determines whether the surveillance video has changed based on the image to be detected and a preset evacuation status library.

[0076] Exemplarily, an evacuation status library is pre-stored in the smart terminal. After obtaining the image to be detected, the smart terminal can determine whether the surveillance video has changed based on the image to be detected and the evacuation status library. When the surveillance video changes, it means that pedestrians or obstructions have appeared in the safety passage.

[0077] S30: When the surveillance video changes, the smart terminal determines whether the change is caused by a pedestrian.

[0078] For example, when the surveillance video changes, the smart terminal will further determine whether the change is caused by a pedestrian, so as to eliminate the impact of passing employees on the detection results as much as possible.

[0079] S40: When the judgment result is no, the intelligent terminal determines that the image to be detected is in a suspected alarm state.

[0080] For example, when the judgment result is no, it means that an obstruction is likely to appear in the safety passage, and therefore, the intelligent terminal determines that the current image to be detected is in a suspected alarm state.

[0081] S50: When N consecutive images to be detected are determined to be in a suspected alarm state, the intelligent terminal marks the area to be detected in the Nth image to be detected to obtain an alarm image.

[0082] Here, N is a preset alarm quantity threshold. For example, N may be a positive integer such as 10, 11, or 12.

[0083] Exemplarily, in order to reduce the possibility of false alarms, an alarm quantity threshold is set. When the number of consecutive images to be detected that are determined to be in a suspected alarm state is less than the alarm quantity threshold, it means that although debris has appeared in the safety passage, the debris has been removed in time. Therefore, the smart terminal will not perform subsequent alarm processing. When N consecutive images to be detected are determined to be in a suspected alarm state, it means that an obstruction is likely to appear in the safety passage, and the obstruction has remained in the safety passage for a long time and has not been removed. Therefore, the smart terminal will mark the area to be detected in the Nth image to be detected, that is, the last image to be detected, and obtain an alarm image. For example, the smart terminal can use white lines to mark the boundary box of the area to be detected in the image to be detected (such as Figure 5 As shown), get the warning image.

[0084] S60: The smart terminal sends an alarm message carrying an alarm image to the cloud server.

[0085] Exemplarily, the smart terminal will send the alarm information carrying the alarm image to the cloud server, so that the cloud server can perform further processing later.

[0086] S70: After receiving the alarm information, the cloud server calls a preset multimodal large model to perform a secondary judgment on the alarm image.

[0087] Exemplarily, after receiving the alarm information, the cloud server can call a preset multimodal large model to perform a secondary judgment on the alarm image to ensure the ultimate accuracy of the safety channel blockage detection method and avoid missed judgments.

[0088] S80: When it is determined that there is an obstacle in the warning image, the cloud server generates and sends warning information to a preset user terminal.

[0089] For example, when the cloud server determines that there is an obstacle in the warning image, it will generate an alarm message and send it to a preset user terminal. The preset user terminal can be a user terminal logged in by a factory administrator account, or a user terminal logged in by a security officer account, or a fixed user terminal for monitoring, etc., to quickly remind the person in charge to deal with the obstruction in the safety passage. In addition, the alarm message can also include the number and location of the source monitoring device of the alarm image.

[0090] Further, such as Figure 2 As shown, before the above S10, the following processing may also be included to complete the configuration of the evacuation state library and the human body feature area in the smart terminal:

[0091] S01: The smart terminal obtains a standard video frame selected by a user from a surveillance video, where the standard video frame is a picture of a safe passage in an unobstructed state in the surveillance video.

[0092] For example, the technician can view the surveillance video through the smart terminal and select several standard video frames from the surveillance video for the smart terminal to record. The standard video frame is an image of a safe passage in a clear state in the surveillance video. For example, it can be an image of a staircase aisle platform in a clear state (such as Figure 3.1 and Figure 3.2 As shown in the figure), images of a safety door in an unobstructed state, etc. Different standard video frames may also be images of the same staircase aisle platform but in different time periods.

[0093] S02: The intelligent terminal crops each standard video frame based on the area to be detected selected by the user, obtains an evacuation status diagram, and generates an evacuation status library.

[0094] For example, the technician can select the area to be monitored on the standard video frame in the smart terminal as the area to be detected. After that, the smart terminal crops each standard video frame according to the area to be detected selected by the user, obtains several evacuation status diagrams, and generates an evacuation status diagram library. For example, when the standard video frame is an image of a stairwell as a safe passage, the cropped evacuation status diagram is an image of the aisle platform in the stairwell (such as Figure 4 shown).

[0095] S03: The intelligent terminal enlarges the area to be detected by a specified multiple with the center point of the area to be detected as the center to obtain a human body feature area.

[0096] Exemplarily, the intelligent terminal can expand the area to be detected by a specified multiple with the center point of the area to be detected as the center to obtain the human feature area. For example, the area to be detected can be expanded by 0.3 times with the center point of the area to be detected as the center to obtain and save the human feature area. For each monitoring video fed back by the monitoring device, a corresponding human feature area will be set.

[0097] Further, see Figure 6 , this step S20 may include the following process:

[0098] S201: Crop the image to be detected according to a preset area to be detected to obtain a partial image to be detected.

[0099] Exemplarily, a detection area for the image to be detected is pre-set in the smart terminal. The smart terminal can crop the image to be detected according to the detection area to obtain a partial image to be detected. For example, the image to be detected may be an image of a stairwell serving as a safety passage. At this time, the detection area is the area where the aisle platform in the stairs in the image is located, and the partial image to be detected obtained after cropping is the image of the aisle platform in the stairs.

[0100] It should be noted that the image to be detected is still retained after cropping, so the image to be detected and the corresponding partial image to be detected will exist at the same time.

[0101] S202: sequentially identifying the similarity between the partial image to be detected and each evacuation state image in a preset evacuation state image library, and taking the maximum similarity as the determination similarity.

[0102] Exemplarily, the size of the local image to be detected must be the same as the size of at least one of the evacuation state diagrams in the evacuation state diagram library. The intelligent terminal compares the similarity between the detected local image and each evacuation state diagram in the evacuation state diagram library in turn, and takes the maximum similarity as the judgment similarity.

[0103] Furthermore, the above-mentioned sequentially identifying the similarity between the partial image to be detected and each evacuation state graph in the preset evacuation state graph library may specifically include the following contents:

[0104] The preset FAST feature point matching algorithm is used to detect and match feature points of the local image to be detected and the target evacuation state map.

[0105] The target evacuation state diagram is any one of all evacuation state diagrams.

[0106] Exemplarily, the intelligent terminal calls a preset FAST feature point matching algorithm to perform feature point detection and matching on the local image to be detected and the target evacuation state diagram until the detection of all evacuation state diagrams and local images to be detected is completed.

[0107] The similarity between the local image to be detected and the target evacuation state map = the number of matched feature points * 2 / the total number of detected feature points.

[0108] S203: If the similarity is determined to be less than a preset similarity threshold, it is determined that the surveillance video has changed.

[0109] Exemplarily, the smart terminal compares the determined similarity with a preset similarity threshold, and when it is identified that the determined similarity is less than the preset similarity threshold, the smart terminal determines that the surveillance video has changed, otherwise it determines that the surveillance video has not changed. For example, the similarity threshold may be 60%.

[0110] Further, step S30 may include the following contents:

[0111] S301: When the judgment result is that the surveillance video has changed, a preset YOLOv5 target detection algorithm is used to identify whether there are pedestrian features in the image to be detected.

[0112] For example, if the judgment result in the above step S20 is that the surveillance video has changed, the smart terminal will call a preset YOLOv5 target detection algorithm, which is pre-trained to be able to recognize pedestrian features. Afterwards, the smart terminal uses the YOLOv5 target detection algorithm to identify whether there are pedestrian features in the image to be detected.

[0113] S302: If there is no pedestrian feature, the judgment result is set to no.

[0114] Exemplarily, when it is recognized that there are no pedestrian features, the intelligent terminal will set the judgment result to no.

[0115] S303: If the recognition result shows that a pedestrian feature exists, the boundary box of the pedestrian feature is compared with a preset human body feature area.

[0116] Exemplarily, when a pedestrian feature is identified, the intelligent terminal compares the bounding box (bbox) of the pedestrian feature with a preset corresponding human feature area.

[0117] S304: When the boundary box of the pedestrian feature and the preset human feature area have an intersection, the judgment result is set to yes.

[0118] Exemplarily, when the boundary box of the pedestrian feature and the preset human feature area intersect, it means that the change of the monitoring video in S20 is caused by a pedestrian passing by, and therefore, the smart terminal sets the judgment result to yes.

[0119] S305: When there is no intersection between the boundary box of the pedestrian feature and the preset human feature area, the judgment result is set to no.

[0120] Exemplarily, when there is no intersection between the boundary box of the pedestrian feature and the preset human feature area, it means that the change of the surveillance video in S20 is not caused by a pedestrian passing by, and therefore, the smart terminal sets the judgment result to no.

[0121] Furthermore, the above S70 may specifically include the following contents:

[0122] S701: After receiving the alarm information, the cloud server calls the preset InternVL multimodal large model.

[0123] Among them, the InternVL multimodal large model has been pre-trained and learned and can be used for subsequent processing.

[0124] S702: The cloud server uses the alarm image in the alarm information as the image input of the InternVL multimodal large model, and calls the preset text information as the prompt text input of the InternVL multimodal large model.

[0125] Exemplarily, the cloud server uses the warning image in the warning information as the image input of the InternVL multimodal large model, and at the same time, calls the preset text information as the prompt text input of the InternVL multimodal large model. For example, the text information can be "Please determine whether there is an obstacle in the area surrounded by the white lines, and directly answer, 'obstacle exists' or 'no obstacle', and skip the cause analysis."

[0126] S703: The cloud server processes the image input and prompt text input respectively through the InternVL multimodal large model to obtain a secondary judgment result for the warning image.

[0127] Exemplarily, the cloud server processes the image input and prompt text input respectively through the InternVL multimodal large model to obtain a secondary judgment result for the warning image.

[0128] Furthermore, in step S703, the cloud server calls the InternVL multimodal large model to perform a text processing flow on the prompt text input to obtain the output of the text processing flow, that is, the text vector. The specific process can be as follows:

[0129] First, the prompt text input is segmented according to the pre-stored vocabulary file. Taking the text information "Please judge whether there are obstacles in the area surrounded by white lines, directly answer, 'obstacles exist' or 'no obstacles exist', skip cause analysis" as an example, the segmentation result can be: "Please / judge / whether / there / are / in / the / area / enclosed / by / white / lines / , / directly / answer / , / obstacles exist / , / or / ' / no / obstacles exist / ', / skip / cause / analysis / . / " Among them, single entries are separated by " / ". After that, each entry after word segmentation is processed by text vectorization to obtain a text vector. For example, if the maximum length of the text is defined as M, the size of the text vector should be 1*M. If the text length is less than M, the rest is filled with empty characters. For example, "please" is converted to "1", "judgment" is converted to "19", "existence" is converted to "888", ..., "obstacle" is converted to "244", "analysis" is converted to "574", and "." is converted to "874", and the space character is 666, then the final output text vector is (0, 19, ..., 888, 244, ..., 574, 874, 666, ..., 666), and the length of the text vector is M.

[0130] Further, in this step S703, the cloud server calls the InternVL multimodal large model to perform an image processing flow on the image input, that is, the alarm image, and obtains the output of the image processing flow, that is, the image feature vector. The specific process can be as follows:

[0131] First, a dynamic high-resolution strategy is used to divide the warning image into several small blocks. For example, the warning image is an image with a resolution of 800×1300. First, its aspect ratio is calculated to be 8:13, and the aspect ratio is compared with several pre-defined preset aspect ratios to obtain a matching preset aspect ratio, such as 2:3. Then, the aspect ratio of the warning image is also adjusted to 2:3. At this time, the resolution of the warning image is adjusted to 896×1344. Then, the adjusted warning image is divided into 6 448×448 image blocks.

[0132] Afterwards, the InternViT-6B visual encoder is used to extract features from the above-mentioned small image blocks to obtain a 6*3*28*28 feature map. Afterwards, the .fallen operation is performed on the above-mentioned feature map to obtain 14112 (i.e. 6*3*28*28) tokens (i.e. pixels). Afterwards, the 14112 tokens are subjected to a pixel shuffle operation to reduce the number of tokens to 1 / 4 of the initial number 14112. At this time, the size of the obtained image feature input is 1*3528. Afterwards, the image feature input is input into the preset MLP Project to align the output of the image processing process with the output of the text processing process, and obtain an image feature vector of size 1*M.

[0133] Furthermore, in this step S703, the cloud server calls the InternVL multimodal large model to concatenate the text vector and the image feature vector to obtain the final feature vector.

[0134] Exemplarily, the InternVL multimodal large model concatenates a text vector of size 1*M and an image feature vector of size 1*M to obtain a final feature vector of size 2*M.

[0135] Furthermore, in step S703, the cloud server calls the InternVL multimodal large model to input the final feature vector into the preset InternL M2 Chat 20B language large model to obtain a secondary judgment result for the warning image.

[0136] Exemplarily, the cloud server calls the InternVL multimodal large model to input the final feature vector into the preset InternL M2 Chat 20B language large model. The output length of the InternL M2 Chat 20B language large model can be defined as N, and a 1*N feature output is obtained. For example, when the InternVL multimodal large model determines that there is an obstacle, the feature output is (888, 244, 666...666). Afterwards, the final Chinese output, i.e., "there is an obstacle", is obtained by combining the vocabulary file and converting it as the secondary judgment result for the warning image, so that the cloud server generates and sends the warning information to the preset user terminal in the subsequent S80.

[0137] The present application provides a method for detecting blockage in a safe passage, which uses an intelligent terminal to detect the latest video frame in the surveillance video, that is, the image to be detected, to determine whether the surveillance video has changed, that is, whether there is a risk of blockage. After judging that there is a risk of blockage, the intelligent terminal further determines whether the change is caused by pedestrians to eliminate the interference of pedestrians on the detection results and improve the accuracy. After eliminating the interference of pedestrian factors, the intelligent terminal will determine that the image to be detected is in a suspected alarm state, that is, it is suspected that a blockage has occurred. When the suspected blockage lasts for a long enough time, the control terminal will further send an alarm message to the cloud server, and then the cloud server will make a final secondary judgment to finally confirm whether the safe passage has been blocked. Through multiple judgments, the detection of blockage in the safe passage can be accurately realized. In addition, in the process of determining whether the image has changed, the detection is performed on the specified area, rather than the detection of the debris itself, so there is no need to exhaustively enumerate the types of blockages, reducing the training cost of the algorithm and avoiding the possibility of insufficient algorithm performance.

[0138] The present application also provides a safety passage blockage detection system, referring to Figure 7 , which includes a smart terminal and a cloud server. Among them. The smart terminal includes:

[0139] The image acquisition module is used to acquire the image to be detected, where the image to be detected is the latest video frame read from the surveillance video.

[0140] The image change determination module is used to determine whether the surveillance video has changed based on the image to be detected and the preset evacuation status library.

[0141] The pedestrian feature determination module is used to determine whether the change is caused by a pedestrian when the surveillance video changes.

[0142] The pedestrian feature determination module is also used to determine that the image to be detected is in a suspected alarm state when the determination result is no.

[0143] The alarm image generation module is used to mark the area to be detected in the Nth image to be detected to obtain an alarm image when N consecutive images to be detected are determined to be in a suspected alarm state, where N is a preset alarm quantity threshold.

[0144] The alarm upload module is used to send alarm information carrying alarm images to a preset cloud server.

[0145] The cloud server includes:

[0146] The judgment module is used to call a preset multimodal large model to perform a secondary judgment on the warning image after receiving the warning information.

[0147] The warning module generates and sends warning information to a preset user terminal when it is determined that there is an obstacle in the warning image.

[0148] refer to Figure 8 The present application also provides an intelligent device, which includes a processor and a memory.

[0149] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire device, and executes various functions of the device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but may be implemented separately through a chip.

[0150] Optionally, the processor implements the safety channel blockage detection method provided by the above-mentioned various method embodiments when executing the program instructions in the memory.

[0151] The memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the device, etc.

[0152] Optionally, the present application also provides a storage medium, which stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the safety channel blockage detection method provided by the above method embodiment.

[0153] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this application.

Claims

1. A safety channel blockage detection method, characterized in that: The method is based on a safety channel blockage detection system, which includes an intelligent terminal and a cloud server, and the intelligent terminal is used to perform the following processing: Acquire an image to be detected, where the image to be detected is the latest video frame read from the surveillance video; Based on the image to be detected and a preset evacuation status library, determining whether the surveillance video has changed; When the surveillance video changes, determining whether the change is caused by a pedestrian; When the judgment result is no, determining that the image to be detected is in a suspected alarm state; When N consecutive images to be detected are determined to be in a suspected alarm state, a detection area is marked in the Nth image to be detected to obtain an alarm image, where N is a preset alarm quantity threshold; Sending the warning information carrying the warning image to the cloud server; The cloud server is used to perform the following processing: After receiving the warning information, calling a preset multi-modal large model to perform a secondary judgment on the warning image; When it is determined that there is an obstacle in the warning image, warning information is generated and sent to a preset user terminal.

2. The method according to claim 1, characterized in that The determining whether the monitoring video has changed based on the image to be detected and a preset evacuation status library includes: According to a preset area to be detected, the image to be detected is cropped to obtain a partial image to be detected; sequentially identifying the similarity between the partial image to be detected and each evacuation state image in a preset evacuation state image library, and taking the maximum similarity as the determination similarity; If the determined similarity is less than a preset similarity threshold, it is determined that the surveillance video has changed.

3. The method according to claim 2, characterized in that The step of sequentially identifying the similarity between the partial image to be detected and each evacuation state diagram in a preset evacuation state diagram library includes: Using a preset FAST feature point matching algorithm to perform feature point detection and matching on the local image to be detected and the target evacuation state diagram, the target evacuation state diagram being any one of all the evacuation state diagrams; The similarity between the local image to be detected and the target evacuation state map = the number of matched feature points*2 / the total number of detected feature points.

4. The method according to claim 1, characterized in that: When the judgment result is that the surveillance video has changed, judging whether the change is caused by a pedestrian includes: When the judgment result is that the surveillance video has changed, the preset YOLOv5 target detection algorithm is used to identify whether there are pedestrian features in the image to be detected; If the pedestrian feature does not exist, the judgment result is set to no; If the recognition result shows that the pedestrian feature exists, the boundary box of the pedestrian feature is compared with a preset human feature area; When the boundary box of the pedestrian feature and the preset human feature area have an intersection, the judgment result is set to yes; When there is no intersection between the boundary box of the pedestrian feature and the preset human feature area, the judgment result is set to no.

5. The method according to claim 2, characterized in that: Before acquiring the image to be detected, the intelligent terminal further performs the following processing: Obtaining a standard video frame selected by a user from a surveillance video, wherein the standard video frame is a picture of a safe passage in an unobstructed state in the surveillance video; Based on the area to be detected selected by the user, each of the standard video frames is cropped to obtain an evacuation state diagram, and the evacuation state diagram library is generated; Taking the center point of the area to be detected as the center, the area to be detected is enlarged by a specified multiple to obtain a human body feature area.

6. The method according to claim 1, characterized in that For the cloud server, after receiving the alarm information, calling a preset multi-modal large model to perform a secondary judgment on the alarm image includes: After receiving the alarm information, calling the preset InternVL multimodal large model; Using the warning image in the warning information as the image picture input of the InternVL multimodal large model, and calling the preset text information as the prompt text input of the InternVL multimodal large model; The image picture input and prompt text input are processed respectively through the InternVL multimodal large model to obtain a secondary judgment result for the warning image.

7. The method according to claim 6, characterized in that In the step of processing the image input and the prompt text input respectively through the InternVL multimodal large model to obtain a secondary judgment result for the warning image, the cloud server calls the InternVL multimodal large model to perform the following processing: Performing a text processing flow on the prompt text input to obtain an output of the text processing flow, i.e., a text vector; Performing an image processing process on the image input, i.e., the alarm image, to obtain an output of the image processing process, i.e., an image feature vector; Concatenate the text vector and the image feature vector to obtain a final feature vector; The final feature vector is input into a preset InternL M2 Chat 20B language model to obtain a secondary judgment result for the warning image.

8. A safety passage blockage detection system, characterized in that: It includes a smart terminal and a cloud server, wherein the smart terminal includes: An image acquisition module is used to acquire an image to be detected, where the image to be detected is the latest video frame read from the surveillance video; An image change determination module, used to determine whether the monitoring video has changed based on the image to be detected and a preset evacuation status library; A pedestrian feature determination module, used to determine whether the change is caused by a pedestrian when the monitoring video changes; The pedestrian feature determination module is further configured to determine that the image to be detected is in a suspected alarm state when the determination result is negative; An alarm image generation module is used to mark the area to be detected in the Nth image to be detected to obtain an alarm image when N consecutive images to be detected are determined to be in a suspected alarm state, where N is a preset alarm quantity threshold; An alarm uploading module is used to send the alarm information carrying the alarm image to a preset cloud server; The cloud server includes: A determination module, configured to call a preset multimodal large model to perform a secondary determination on the warning image after receiving the warning information; The warning module generates and sends warning information to a preset user terminal when it is determined that there is an obstacle in the warning image.

9. A smart device, characterized in that: The smart device includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium having a program stored thereon, wherein when the program is executed by a processor, the program is used to implement the method according to any one of claims 1 to 7.

Images