Security detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a security detection method and device, electronic equipment and a storage medium, and the method comprises the steps: receiving the text information of a security detection rule inputted by a front-end interface, and collecting a to-be-detected image; and inputting the text information and the to-be-detected image into the pre-trained multi-modal neural network model, and determining the target object meeting the security detection rule in the to-be-detected image based on the multi-modal neural network model, so that compared with rule limitation and model training for each detection scene, the security detection efficiency of the to-be-detected image is improved. According to the method and the device, security detection of different scenes can be carried out only by changing the text information input by the front-end interface, so that security detection of different scenes by using the same model is realized.

Description

Technical Field

[0001] The present application relates to the field of security detection technology, and in particular to a security detection method, device, electronic device and storage medium. Background Art

[0002] In the field of production safety, dangerous operations, special operations and special operations are three typical key operations, which are the high-incidence links of production safety accidents. Therefore, it is of great significance to strengthen the safety supervision and inspection of these three types of operations. The traditional safety supervision method usually sets up cameras at the work site, detects safety-related objects such as helmets, escalators and seat belts in real time through the target detection model, and uses the fixed rules pre-set in the background to judge whether there are any illegal operations in real time, and promptly alarm and rectify them.

[0003] However, the traditional method of adding deterministic rules to target detection has a limited application scope. For example, the target detection model is customized for a certain scene. When the scene changes, the detected object changes, and the target detection model needs to be retrained. It is difficult to achieve generalization in multiple scenes, that is, different models need to be trained for different scenes to achieve security detection. Therefore, how to use the same model for security detection in different scenes is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a security detection method, device, electronic device and storage medium to solve the problem of how to use the same model to perform security detection on different scenarios.

[0005] In a first aspect, a method for security detection is provided, the method comprising:

[0006] Receive the text information of the security detection rules inputted from the front-end interface and collect the images to be detected;

[0007] Inputting the text information and the image to be detected into a pre-trained multimodal neural network model, and determining the target object in the image to be detected that meets the security detection rule based on the multimodal neural network model;

[0008] Among them, the multimodal neural network model is used to extract features from the text information to obtain a first text feature vector; to extract features from the image to be detected to obtain a first image feature vector; to determine a first fused feature vector based on the first text feature vector and the first image feature vector; and to determine a target object in the image to be detected that meets the security detection rule based on the first fused feature vector.

[0009] Furthermore, the multimodal neural network model includes: a text encoder, a visual encoder, a feature enhancer, and a target decoder;

[0010] The process of determining the target object satisfying the safety detection rule in the image to be detected based on the multimodal neural network model includes:

[0011] Extracting features from the text information based on the text encoder to obtain the first text feature vector;

[0012] Extracting features of the image to be detected based on the visual encoder to obtain the first image feature vector;

[0013] Inputting the first text feature vector and the first image feature vector into the feature enhancer, and determining the first fused feature vector based on the feature enhancer;

[0014] The first fused feature vector is input into the target decoder, and a target object satisfying the security detection rule in the image to be detected is determined based on the target decoder.

[0015] Furthermore, before determining the target object satisfying the safety detection rule in the image to be detected based on the multimodal neural network model, the method further includes:

[0016] Determine first subtext information corresponding to each object in the text information;

[0017] For each first subtext information, determine whether the first subtext information exists in a pre-stored detection object text library; if not, obtain a first image corresponding to the first subtext information from a pre-stored text image library;

[0018] The process of determining the target object in the image to be detected that satisfies the security detection rule includes:

[0019] The first image, the text information and the image to be detected are input into the multimodal neural network model, and based on the multimodal neural network model, a target object in the image to be detected that meets the security detection rule is determined.

[0020] Furthermore, the multimodal neural network model further comprises: a visual cue encoder;

[0021] The process of determining the target object in the image to be detected that satisfies the security detection rule includes:

[0022] Extracting features from the first image based on the visual cue encoder to obtain a second image feature vector;

[0023] Inputting the first text feature vector, the first image feature vector and the second image feature vector into the feature enhancer, and determining the first fused feature vector based on the feature enhancer;

[0024] The first fused feature vector is input into the target decoder, and a target object satisfying the security detection rule in the image to be detected is determined based on the target decoder.

[0025] Furthermore, the process of pre-training the multimodal neural network model includes:

[0026] Obtaining a first sample image in a first training set and sample text information of a security detection rule corresponding to the first sample image;

[0027] Inputting the sample text information into the text encoder to obtain a second text feature vector; inputting the first sample image into the visual encoder to obtain a third image feature vector;

[0028] Inputting the second text feature vector and the third image feature vector into the feature enhancer, and determining a second fused feature vector based on the feature enhancer;

[0029] Inputting the second fused feature vector into the target decoder, and determining predicted position information of the target object satisfying the safety detection rule in the first sample image based on the target decoder;

[0030] Determine a first loss value based on the predicted position information and the annotated position information of the target object that meets the safety detection rule in the first sample image; and train the multimodal neural network model based on the first loss value.

[0031] Furthermore, the process of obtaining the sample text information includes:

[0032] The first sample image is input into a pre-trained text description model, and sample text information of a security detection rule corresponding to the first sample image is determined based on the text description model.

[0033] Furthermore, before obtaining the first sample image in the first training set and the sample text information of the security detection rule corresponding to the first sample image, the method further includes:

[0034] Obtaining a second sample image and a category label corresponding to the detection object in the second training set;

[0035] Inputting the category label into the text encoder to extract a third text feature vector of the category label; inputting the second sample image into the visual encoder to extract a fourth image feature vector of the second sample image;

[0036] Inputting the third text feature vector and the fourth image feature vector into the feature enhancer, and obtaining a third fused feature vector of the category label and the second sample image based on an attention extraction mechanism;

[0037] Inputting the third fused feature vector into the target decoder to determine the predicted category of the detected object in the second sample image;

[0038] A second loss value is determined according to the predicted category and the category label; and the feature enhancer and the target decoder in the multimodal neural network model are trained according to the second loss value.

[0039] In a second aspect, a security detection device is provided, the device comprising:

[0040] The acquisition module is used to receive the text information of the security detection rules inputted from the front-end interface and collect the image to be detected;

[0041] A processing module is used to input the text information and the image to be detected into a pre-trained multimodal neural network model, and determine the target object in the image to be detected that meets the security detection rule based on the multimodal neural network model; wherein the multimodal neural network model is used to perform feature extraction on the text information to obtain a first text feature vector; perform feature extraction on the image to be detected to obtain a first image feature vector; determine a first fused feature vector based on the first text feature vector and the first image feature vector; and determine the target object in the image to be detected that meets the security detection rule based on the first fused feature vector.

[0042] Furthermore, the multimodal neural network model includes: a text encoder, a visual encoder, a feature enhancer, and a target decoder; the processing module is specifically used to perform feature extraction on the text information based on the text encoder to obtain the first text feature vector; perform feature extraction on the image to be detected based on the visual encoder to obtain the first image feature vector; input the first text feature vector and the first image feature vector into the feature enhancer, and determine the first fused feature vector based on the feature enhancer; input the first fused feature vector into the target decoder, and determine the target object in the image to be detected that meets the security detection rule based on the target decoder.

[0043] Furthermore, the processing module is also used to determine the first sub-text information corresponding to each object in the text information; for each first sub-text information, determine whether the first sub-text information exists in a pre-saved detection object text library; if not, obtain the first image corresponding to the first sub-text information from a pre-saved text image library; input the first image, the text information and the image to be detected into the multimodal neural network model, and determine the target object in the image to be detected that meets the security detection rules based on the multimodal neural network model.

[0044] Furthermore, the multimodal neural network model also includes: a visual cue encoder; the processing module is specifically used to extract features of the first image based on the visual cue encoder to obtain a second image feature vector; the first text feature vector, the first image feature vector and the second image feature vector are input into the feature enhancer, and the first fused feature vector is determined based on the feature enhancer; the first fused feature vector is input into the target decoder, and the target object in the image to be detected that meets the security detection rule is determined based on the target decoder.

[0045] Furthermore, the acquisition module is specifically used to acquire a first sample image in a first training set and sample text information of a security detection rule corresponding to the first sample image;

[0046] The processing module is specifically used to input the sample text information into the text encoder to obtain a second text feature vector; input the first sample image into the visual encoder to obtain a third image feature vector; input the second text feature vector and the third image feature vector into the feature enhancer, and determine a second fused feature vector based on the feature enhancer; input the second fused feature vector into the target decoder, and determine the predicted position information of the target object in the first sample image that meets the safety detection rule based on the target decoder; determine a first loss value based on the predicted position information and the labeled position information of the target object in the first sample image that meets the safety detection rule; and train the multimodal neural network model based on the first loss value.

[0047] Furthermore, the processing module is specifically used to input the first sample image into a pre-trained text description model, and determine sample text information of the security detection rule corresponding to the first sample image based on the text description model.

[0048] Furthermore, the acquisition module is further used to acquire a second sample image and a category label corresponding to the detection object in the second training set;

[0049] The processing module is also used to input the category label into the text encoder and extract the third text feature vector of the category label; input the second sample image into the visual encoder and extract the fourth image feature vector of the second sample image; input the third text feature vector and the fourth image feature vector into the feature enhancer, and based on the attention extraction mechanism, obtain the third fused feature vector of the category label and the second sample image; input the third fused feature vector into the target decoder to determine the predicted category of the detection object in the second sample image; determine a second loss value based on the predicted category and the category label; and train the feature enhancer and the target decoder in the multimodal neural network model based on the second loss value.

[0050] According to a third aspect, an electronic device is provided, the electronic device comprising a processor, the processor being configured to implement the steps of any of the security detection methods described above when executing a computer program stored in a memory.

[0051] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of any of the security detection methods described above are implemented.

[0052] In a fifth aspect, a computer program product is provided, the computer program product comprising: a computer program code, when the computer program code is run on a computer, the computer executes the steps of any of the security detection methods described above.

[0053] In an embodiment of the present application, text information of security detection rules inputted by a front-end interface is received, and an image to be detected is collected; the text information and the image to be detected are inputted into a pre-trained multimodal neural network model, and based on the multimodal neural network model, the target object in the image to be detected that meets the security detection rules is determined. Compared with defining rules and training models for each detection scenario, the present application realizes that security detection of different scenarios can be performed by only changing the text information inputted by the front-end interface, thereby realizing security detection of different scenarios using the same model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1A schematic diagram of a process of a security detection method provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of the structure of a multimodal neural network model provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of a specific process of determining a target object that meets a safety detection rule in an image to be detected by a multimodal neural network model provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of a specific process of text information detection of a security detection rule provided in an embodiment of the present application;

[0059] Figure 5 A complete structural diagram of a multimodal neural network model provided in an embodiment of the present application;

[0060] Figure 6 A schematic diagram of a process for determining a target object that satisfies a safety detection rule in an image to be detected by a multimodal neural network model including a first image provided in an embodiment of the present application;

[0061] Figure 7 One of the schematic diagrams of a process for training a multimodal neural network model provided in an embodiment of the present application;

[0062] Figure 8 A second schematic diagram of a process for training a multimodal neural network model provided in an embodiment of the present application;

[0063] Fig. 9 A system schematic diagram of a safety detection method provided in an embodiment of the present application;

[0064] Fig.10 A schematic diagram of the structure of a safety detection device provided in an embodiment of the present application;

[0065] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0067] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0068] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0069] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0070] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0072] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0073] Embodiment 1:

[0074] Figure 1 A schematic diagram of a process of a security detection method provided in an embodiment of the present application, the process includes the following steps:

[0075] S101: receiving text information of security detection rules inputted from the front-end interface, and collecting images to be detected.

[0076] A security detection method provided in an embodiment of the present application is applied to an electronic device, which may be a computer (Personal Computer, PC), a server, etc.

[0077] The text information of the safety detection rules refers to the text information of the safety detection rules set by the user for the detection scenario. For example, for the construction site operation scenario, the text information of the safety detection rules is "a person wearing work clothes but not a safety helmet"; for the high-altitude operation scenario, the text information of the safety detection rules is "a person not wearing a safety belt in the crane tower", etc.

[0078] The text information of the security detection rules can be input by the user through a handheld electronic device, such as the user's mobile phone, which has an application for security detection installed on it, and input through the front-end interface of the application, or it can be input by the user through the front-end interface of an electronic device deployed in the detection scenario.

[0079] For frequently used detection scenarios, the text information of the safety detection rule can be the text information of the safety detection rule set by the user for the detection scenario, and when the detection scenario changes, it is changed according to actual needs. For example, for the construction site operation scenario, the text information of the safety detection rule pre-set by the user is "a person wearing work clothes without a safety helmet". When the detection scene changes to a high-altitude operation scene, according to the needs of the high-altitude operation scene, the safety detection rule is changed to "a person in a hanging tower without a safety belt". At this time, the user can delete the text information of "a person wearing work clothes without a safety helmet" and input the new text information of "a person in a hanging tower without a safety belt". Optionally, the user can directly input the new text information of "a person in a hanging tower without a safety belt". At this time, the initially set text information of "a person wearing work clothes without a safety helmet" automatically becomes invalid.

[0080] In a possible implementation, in the same detection scenario, the correspondence between the text information of the safety detection rule and the time period can be set, that is, different time periods correspond to different text information of the safety detection rule. For example, for a construction site operation scenario, between 9:00 and 23:00 of the same day, the text information of the safety detection rule set is "a person wearing work clothes but not a helmet"; between 23:00 of the same day and 9:00 of the next day, the text information of the safety detection rule set is "a person not wearing reflective clothing".

[0081] For the same detection scenario, the user can set the text information of one security detection rule or the text information of multiple security detection rules.

[0082] The image to be detected refers to the image of the operation that needs to be tested for safety. Optionally, the image to be detected is obtained by using a high-resolution camera to collect images of the detection scene. The image to be detected can be an image collected at a fixed time period, for example, an image is collected every 30 seconds, and the collected image is used as the image to be detected for detection; or the camera can collect a video of the detection scene, and the electronic device uses each frame of the collected video as the image to be detected for detection.

[0083] S102: Input text information and an image to be detected into a pre-trained multimodal neural network model, and determine the target object in the image to be detected that meets the security detection rules based on the multimodal neural network model; wherein the multimodal neural network model is used to perform feature extraction on the text information of the security detection rules to obtain a first text feature vector corresponding to the text information of the security detection rules; perform feature extraction on the image to be detected to obtain a first image feature vector of the image to be detected; determine a first fused feature vector based on the first text feature vector and the first image feature vector; and determine the target object in the image to be detected that meets the security detection rules based on the first fused feature vector.

[0084] The received text information of the safety detection rules and the collected images to be detected are respectively input into the pre-trained multimodal neural network model. The multimodal neural network model is used to determine the target object that meets the safety detection rules in the image to be detected. For example, the target object can be a person or a car.

[0085] Optionally, before inputting the image to be detected into the multimodal neural network model, the image to be detected may be subjected to image preprocessing, exemplarily, image enhancement: the image to be detected is subjected to enhancement processing to improve image quality and contrast and eliminate interference such as noise.

[0086] In an embodiment of the present application, text information of security detection rules inputted by a front-end interface is received, and an image to be detected is collected; the text information and the image to be detected are inputted into a pre-trained multimodal neural network model, and based on the multimodal neural network model, the target object in the image to be detected that meets the security detection rules is determined. Compared with defining rules and training models for each detection scenario, the present application realizes that security detection of different scenarios can be performed by only changing the text information inputted by the front-end interface, thereby realizing security detection of different scenarios using the same model.

[0087] Embodiment 2:

[0088] For ease of understanding, the embodiments of the present application fully illustrate the specific process of using a multimodal neural network model to determine a target object that meets the security detection rules in an image to be detected.

[0089] Figure 2 A schematic diagram of the structure of a multimodal neural network model provided in an embodiment of the present application. Figure 2 As shown, the multimodal neural network model of the present application includes: a text encoder, a visual encoder, a feature enhancer, and a target decoder. Figure 3 A schematic diagram of a specific process of determining a target object that meets a security detection rule in an image to be detected by a multimodal neural network model provided in an embodiment of the present application, such as Figure 3As shown, the process includes the following steps:

[0090] S301: Extract features from text information based on a text encoder to obtain a first text feature vector.

[0091] The text encoder is used to extract the first text feature vector of the text information of the safety detection rule inputted into the front-end interface, for example, the first text feature vector corresponding to "a person wearing work clothes but not a safety helmet".

[0092] The text encoder performs word segmentation on the text information of the input safety detection rules and converts the input text information into single words. Through the vocabulary data of the text encoder, the single word is converted into a series of IDs, for example, "safety helmet" is converted into "00000001". The ID information is input into the deep bidirectional language representation model (BERT) of the text encoder for feature extraction to obtain the first text feature vector. The first text feature vector can express the semantic information of the text information of the user input safety detection rules.

[0093] In a possible implementation, the text encoder may include a pre-trained language model (T5). After obtaining a series of IDs for single word conversion, the ID information is input into the pre-trained language model (T5) of the text encoder for feature extraction to obtain a first text feature vector.

[0094] S302: Extract features of the image to be detected based on the visual encoder to obtain a first image feature vector.

[0095] The visual encoder is used to extract the feature vector containing the image semantic information of the collected image to be detected. The image to be detected is divided into small blocks, and the feature information of each small block of the image to be detected is extracted; the feature information of each small block is extracted to obtain the first image feature vector of the image to be detected, that is, the overall image feature information of the image to be detected is extracted by the pre-trained visual encoder, that is, the first image feature vector. The first image feature vector can express the image semantic information of the collected image to be detected.

[0096] S303: Input the first text feature vector and the first image feature vector into a feature enhancer, and determine a first fused feature vector based on the feature enhancer.

[0097] The feature enhancer is used to fuse features from different sources, and integrate different input features into a unified representation space through semantic alignment and feature mapping. Specifically, the feature enhancer aligns the text semantic information of the text information, i.e., the first text feature vector, and the image semantic information of the image to be detected, i.e., the first image feature vector, and maps the first text feature vector and the first image feature vector, so as to unify the text information and the object information in the image to be detected. In other words, the feature enhancer fuses the first text feature vector and the first image feature vector into a first fused feature vector. Optionally, the first text feature vector and the first image feature vector are spliced ​​to obtain a first fused feature vector. The first fused feature vector can express the information mapped between the text information of the security detection rule and the image to be detected. In this case, the first fused feature vector contains the first text feature vector of the text information and the first image feature vector of the image to be detected.

[0098] S304: Input the first fused feature vector to a target decoder, and determine a target object that satisfies a security detection rule in the image to be detected based on the target decoder.

[0099] The target decoder is used to perform reasoning based on the feature vector after cross-modal fusion, determine the target object in the image that meets the safety detection rules, and output the detected target object corresponding to the sentence of the user input safety detection rule. Optionally, the multimodal neural network model can also output the category name and corresponding rectangular frame coordinates of each object.

[0100] The feature vector after cross-modal fusion refers to the first fused feature vector after the two different modal types of text in step 301 and image in step 302 are fused.

[0101] The safety detection rule contains multiple objects, and the target object is determined from the multiple objects by the target decoder. For example, if the text information of the safety detection rule is "a person wearing work clothes but not a helmet", then the multiple objects contained in the safety detection rule are "work clothes", "helmet" and "person", and the target object is "person".

[0102] The multimodal neural network model outputs the detection target object corresponding to the sentence of the user input safety detection rule. For example, the safety detection rule input by the user is "a person wearing work clothes but not a helmet". After passing through the multimodal neural network model, the output is an image marked with "a person wearing work clothes but not a helmet".

[0103] Optionally, the multimodal neural network model may also output a category name of each object and a corresponding rectangular frame coordinate. The category name of each object refers to a person or an object, etc. The rectangular frame coordinate may be a specific location of the output, or may be marked with a rectangular frame in the image. For example, the safety detection rule input by the user is "a person wearing work clothes but not a safety helmet". After passing through the multimodal neural network model, in addition to outputting an image marked with "a person wearing work clothes but not a safety helmet", optionally, the category names of multiple objects in the safety detection rule can also be output, for example, "work clothes", "safety helmet", "person"; and the corresponding rectangular box coordinates of each object are output accordingly. For example, taking the lower left corner of the detection image as the origin, the vertex coordinates of "work clothes" in the detection image are (a1, b1), (a2, b2), (a3, b3) and (a4, b4); the vertex coordinates of "safety helmet" in the detection image are (c1, d1), (c2, d2), (c3, d3) and (c4, d4); the vertex coordinates of "person" in the detection image are (e1, f1), (e2, f2), (e3, f3) and (e4, f4).

[0104] In a possible implementation, for the detection scene, the user may input a specific detection object. For example, for the detection scene where traffic is prohibited, the user may input text information of the safety detection objects of "people" and "cars". At this time, the multimodal neural network model will input target images marked with "people" and "cars" for the collected images. Optionally, the object category names "people" and "cars" and the corresponding specific coordinates are also output. For example, the vertex coordinates of "people" in the target image are (g1, h1), (g2, h2), (g3, h3) and (g4, h4); the vertex coordinates of "cars" in the target image are (p1, q1), (p2, q2), (p3, q3) and (p4, q4).

[0105] In an embodiment of the present application, the text information of the security detection rule is input into a text encoder to extract the first text feature vector of the text information; the image to be detected is input into a visual encoder to extract the first image feature vector of the image to be detected; the first text feature vector and the first image feature vector are fused based on a feature enhancer to obtain a first fused feature vector, which integrates the text information of the security detection rule and the image information of the image to be detected, and the fused first fused feature vector is extracted by an attention mechanism through a target decoder to determine the target object that meets the security detection rule in the image to be detected, thereby achieving security detection of the detection scene. Compared with the problem that deterministic rule development is time-consuming and cannot cover all scenarios, resulting in a large number of false positives, the present application performs security supervision by inputting security detection rules in the form of text information, which is more convenient to operate and can quickly respond to different rule requirements for different detection scenarios.

[0106] Embodiment 3:

[0107] In order to improve the accuracy of security detection, on the basis of the above embodiments, in an embodiment of the present application, based on the multimodal neural network model, before determining the target object that meets the security detection rules in the image to be detected, the text information of the security detection rules is also detected to improve the accuracy of the multimodal neural network model in identifying the target object. Figure 4 A schematic diagram of a specific process of text information detection of a security detection rule provided in an embodiment of the present application, such as Figure 4 As shown, the process includes the following steps:

[0108] S401: Determine first sub-text information corresponding to each object in the text information.

[0109] After receiving the text information of the safety detection rules inputted from the front-end interface, the electronic device performs word segmentation processing on the text of the safety detection rules to determine the text information corresponding to each object in the text information, that is, the first sub-text information corresponding to each object. For example, the text information of the safety detection rules inputted from the front-end interface is "a person wearing work clothes but not a helmet", and the text encoder performs word segmentation processing on the text of "a person wearing work clothes but not a helmet", and obtains the first sub-text information of "work clothes", "helmet" and "person".

[0110] S402: For each first subtext information, determine whether the first subtext information exists in a pre-stored detection object text library.

[0111] If not, execute S403; if so, execute S405.

[0112] The first sub-text information is the text information corresponding to each object in the text information of the above security detection rule.

[0113] The pre-saved detection object text library contains text information corresponding to the detection object used in the training model. If the first sub-text information can be identified in the detection object text library, that is, the detection object text library contains the first sub-text information, it proves that the object corresponding to the first sub-text information is the detection object used in the training model, and the trained multimodal neural network model can identify the detection object; if the first sub-text information cannot be identified in the detection object text library, that is, the detection object text library does not contain the first sub-text information, it proves that the object corresponding to the first sub-text information is not the detection object used in the training model, and the trained multimodal neural network model cannot identify the detection object.

[0114] S403: Acquire a first image corresponding to the first subtext information from a pre-stored text image library.

[0115] The pre-stored text image library contains text information and images corresponding to the text information. Based on the text image library, the first image corresponding to the first sub-text information is obtained.

[0116] S404: Input the first image, text information and the image to be detected into the multimodal neural network model, and determine the target object in the image to be detected that meets the security detection rules based on the multimodal neural network model.

[0117] The text information of the security detection rules input into the front-end interface, the first image obtained in step 403, and the image to be detected are respectively input into the multimodal neural network model, and the multimodal neural network model extracts corresponding features from the text information, the first image, and the image to be detected, and uses the cross-attention mechanism through the feature enhancer to fuse the features of these three sources, enhance the expression ability of each feature, and obtain a first fused feature vector. Based on the first fused feature vector, the target object that meets the security detection rules in the image to be detected is determined.

[0118] S405: Input the text information and the image to be detected into the multimodal neural network model, and determine the target object in the image to be detected that meets the security detection rules based on the multimodal neural network model.

[0119] The process of inputting text information and the image to be detected into the multimodal neural network model, and determining the target object in the image to be detected that meets the security detection rules based on the multimodal neural network model is the process introduced in the above embodiment 2, which will not be repeated here.

[0120] In an embodiment of the present application, based on a pre-saved text library of detection objects, it is determined whether the objects of the first sub-text information corresponding to the text information of the security detection rules input into the front-end interface are all the detection objects used when training the model; if not, for the first sub-text information that does not exist, the first image corresponding to the first sub-text information is obtained from a pre-saved text image library, and the first image, text information and image to be detected are input into a multimodal neural network model, and based on the multimodal neural network model, the target object in the image to be detected that meets the security detection rules is determined, so that the multimodal neural network model accurately identifies the target object, thereby improving the accuracy of model recognition.

[0121] Embodiment 4:

[0122] For ease of understanding, the present embodiment fully describes the process of step 404. Figure 5 A complete structural diagram of a multimodal neural network model provided in an embodiment of the present application. Figure 5 As shown, the multimodal neural network model also includes: a visual cue encoder. Figure 6A schematic diagram of a process of determining a target object that satisfies a security detection rule in an image to be detected by a multimodal neural network model including a first image provided in an embodiment of the present application, such as Figure 6 As shown, the process includes the following steps:

[0123] S601: extract features from the first image based on a visual cue encoder to obtain a second image feature vector.

[0124] The visual cue encoder is used to guide the multimodal neural network model to detect specific target objects. The first image obtained in step 403 is input to the visual cue encoder, and based on the visual cue encoder, the second image feature vector of the first image is determined, that is, the image corresponding to the detection object used when training the model in the text information of the security detection rule is not input to the visual cue encoder, so that the multimodal neural network model can accurately identify certain professional category equipment that has not been trained.

[0125] The visual cue encoder adopts the same device architecture as the visual encoder in the above-mentioned embodiment 2. Specifically, the visual cue encoder divides the first image into small blocks, extracts feature information of each small block of the first image, and extracts features from the feature information of each small block to obtain a second image feature vector of the first image, that is, extracts the overall image feature information of the first image through the pre-trained visual cue encoder, that is, the second image feature vector. The second image feature vector can express the image semantic information of the first image.

[0126] For example, the real-time dynamic measurement device (RTK device) is not covered during the training of the multimodal neural network model, which makes it difficult for the multimodal neural network model to determine the RTK device only through text information. The present application inputs the first image corresponding to the RTK device into the visual prompt encoder to guide the multimodal neural network model to detect the RTK device, so that the multimodal neural network model can accurately identify the RTK device.

[0127] S602: Input the first text feature vector, the first image feature vector, and the second image feature vector into a feature enhancer, and determine a first fused feature vector based on the feature enhancer.

[0128] The feature enhancer uses a cross-attention mechanism to fuse features from other sources to enhance the expressiveness of each feature, that is, the feature enhancer can fuse the first text feature vector, the first image feature vector, and the second image feature vector to obtain a first fused feature vector of the text information, the image to be detected, and the first image. Optionally, the first text feature vector, the first image feature vector, and the second image feature vector are concatenated to obtain a first fused feature vector. In this case, the first fused feature vector includes the first text feature vector of the text information, the first image feature vector of the image to be detected, and the second image feature vector of the first image.

[0129] S603: Input the first fused feature vector to a target decoder, and determine a target object that satisfies a security detection rule in the image to be detected based on the target decoder.

[0130] The process is consistent with step 304 in the above embodiment 2 and will not be repeated here.

[0131] Embodiment 5:

[0132] In order to achieve safe detection of detection scenarios, this application pre-trains a multimodal neural network model that can determine the target object. Figure 7 One of the schematic diagrams of a process for training a multimodal neural network model provided in an embodiment of the present application, the process includes the following steps:

[0133] S701: Obtain a first sample image in a first training set and sample text information of a security detection rule corresponding to the first sample image.

[0134] The first training set refers to an existing target detection alarm database, and the first sample image refers to each detection alarm image contained in the existing target detection alarm database.

[0135] The first sample image is input into the pre-trained text description model to obtain sample text information of the safety detection rule corresponding to the first sample image, for example, "workers wearing work clothes but not helmets", "someone on the ladder but no one holding it", "ladder set up in a hanging basket" and other sample text information.

[0136] S702: Input the sample text information into the text encoder to obtain a second text feature vector; input the first sample image into the visual encoder to obtain a third image feature vector.

[0137] The text encoder performs word segmentation processing on the input sample text information and converts the input sample text information into a single word. Through the vocabulary data of the text encoder, the single word is converted into a series of IDs. The ID information is input into the deep bidirectional language representation model (BERT) of the text encoder for feature extraction to obtain the second text feature vector of the sample text information. The second text feature vector can express the semantic information of the sample text information input by the user.

[0138] In a possible implementation, the ID information is input into a pre-trained language model (T5) of a text encoder for feature extraction to obtain a second text feature vector of the sample text information.

[0139] The visual encoder divides the first sample image into small blocks, extracts feature information of each small block of the first sample image, and extracts features from the feature information of each small block to obtain a third image feature vector of the first sample image, that is, the overall image feature information of the first sample image is extracted by the visual encoder, that is, the third image feature vector. The third image feature vector can express the image semantic information of the first sample image.

[0140] S703: Input the second text feature vector and the third image feature vector into a feature enhancer, and determine a second fused feature vector based on the feature enhancer.

[0141] The feature enhancer fuses the second text feature vector and the third image feature vector into a second fused feature vector. Optionally, the second text feature vector and the third image feature vector are concatenated to obtain a second fused feature vector. The second fused feature vector can express information mapped between the sample text information and the first sample image.

[0142] S704: Input the second fused feature vector into the target decoder, and determine the predicted position information of the target object that meets the safety detection rule in the first sample image based on the target decoder.

[0143] The second fused feature vector obtained by fusing the second text feature vector of the sample text information and the third text feature vector of the first sample image is input into the target decoder, and the predicted position information of the target object that meets the security detection rule in the first sample image is determined based on the target decoder.

[0144] The predicted position information of the target object refers to the coordinate information of each vertex of the target object in the sample image predicted by the target decoder that meets the safety detection rule. For example, the sample text information is "a person wearing work clothes but not a helmet", that is, the safety detection rule is "a person wearing work clothes but not a helmet", at this time, the target decoder predicts that the target object is "a person" and the vertex coordinates of the "person" in the first sample image are (m1, n1), (m2, n2), (m3, n3) and (m4, n4).

[0145] S705: Determine a first loss value based on the predicted position information and the labeled position information of the target object that meets the safety detection rule in the first sample image; and train the multimodal neural network model based on the first loss value.

[0146] The annotated position information of the target object satisfying the safety detection rule in the first sample image refers to the coordinate positions of each vertex of the target object corresponding to the sample text information determined in the sample image.

[0147] According to the predicted position information and the annotated position information of the target object that meets the safety detection rules in the first sample image, a first loss value is calculated based on the cross entropy loss function, and the model parameters of the multimodal neural network model are adjusted according to the first loss value. Through the input of multiple sample text information and the first sample image, the multimodal neural network model is iteratively trained, and when the first loss value reaches a preset target loss value, a trained multimodal neural network model is obtained. Or when the number of training times meets the preset target value, a trained multimodal neural network model is obtained.

[0148] In an embodiment of the present application, the multimodal neural network model that has been trained can directly change the text information of the input safety detection rules to realize safety detection in different scenarios by only changing the text information input in the front-end interface, thereby realizing generalization of multiple scenarios. That is, the multimodal neural network model after training can recognize a large number of objects based on text information, without the need for additional rule development, and can describe the target objects to be detected using only text information, so that it can be directly applied to dangerous operation detection in multiple scenarios.

[0149] Embodiment 6:

[0150] For ease of understanding, the embodiment of the present application fully explains the process of obtaining sample text information in the above step 701.

[0151] The first sample image is input into a pre-trained text description model, and sample text information of the security detection rule corresponding to the first sample image is determined based on the text description model.

[0152] The pre-trained text description model can be a large multimodal model, such as the large multimodal model GPT-4o or the large-scale visual language model Qwen-VL. The first sample image is input into the text description model to obtain the sample text information of the security detection rule corresponding to the first sample image. Among them, the large multimodal model GPT-4o or the large-scale visual language model Qwen-VL can input images, text and audio, and output any combination of text, audio and images to realize the processing and understanding of multimodal information.

[0153] Specifically, by inputting the first sample image into the text description model, text information describing the attributes of the object is constructed. Construction of text information describing the attributes of the object: Using the image category label information in the target detection dataset, for example, "cat", "dog", through prompt words such as "please describe the cat in the image in detail, including color, position and other information", the text description model is used to output a detailed description, such as "There is a gray cat in the picture, lying on a seat"; for another example, "safety helmet", "work clothes", "person", through prompt words such as "please describe the person in the image in detail, including safety helmet, work clothes and other information", the text description model is used to output a detailed description, such as "a person wearing work clothes but not a safety helmet". Through the text description model, text information describing the attributes of the object can be obtained.

[0154] Specifically, by inputting the first sample image and the data of the existing safety detection rules into the text description model, the text information describing the safety detection rules is constructed. Combined with the data of the existing safety detection rules: using the existing target detection alarm data, such as the first sample images corresponding to "no safety helmet alarm" and "no escalator alarm" respectively; the first sample image and the corresponding specific safety detection rules are input as text into the text description model to construct text information describing the safety detection rules, such as "a person wearing work clothes but not a safety helmet", "someone on the ladder but no one holding the ladder", "a ladder set up in a hanging basket", so as to construct data containing safety detection rules.

[0155] By using text information describing the attributes of objects and data combined with safety detection rules to train the multimodal neural network model, the multimodal neural network model can understand the attributes of objects and the positional relationship between objects. After training, the multimodal neural network model can recognize a large number of objects based on text information, without the need for additional rule development. Only text information can be used to describe the target object to be detected, and it can be directly applied to dangerous operation detection in multiple scenarios.

[0156] Embodiment 7:

[0157] In order to improve the accuracy of the multimodal neural network model in identifying the detection object, the present application also trains the multimodal neural network model in identifying the detection object before obtaining the first sample image in the first training set and the sample text information of the security detection rule corresponding to the first sample image, so as to improve the accuracy of the multimodal neural network model in identifying the detection object. Figure 8 A second schematic diagram of a process for training a multimodal neural network model provided in an embodiment of the present application includes the following steps:

[0158] S801: Obtain a second sample image and a category label corresponding to a detection object in a second training set.

[0159] The second training set refers to an existing target detection data set, which contains images corresponding to the detection object and category labels corresponding to the detection object. Based on the target detection data set, the second sample image and category label corresponding to the detection object are obtained. For example, if the detection object is a "ladder", the second sample image corresponding to the "ladder" and the category label "ladder" are obtained; if the detection object is a "safety helmet", the second sample image corresponding to the "safety helmet" and the category label "safety helmet" are obtained.

[0160] S802: Input the category label into the text encoder to extract the third text feature vector of the category label; input the second sample image into the visual encoder to extract the fourth image feature vector of the second sample image.

[0161] The category label is input into the text encoder, and feature extraction is performed based on the deep bidirectional language representation model (BERT) of the text encoder to obtain the third text feature vector of the category label.

[0162] In a possible implementation, the category label is input into a text encoder, and feature extraction is performed based on a pre-trained language model (T5) of the text encoder to obtain a third text feature vector of the category label.

[0163] The visual encoder divides the second sample image into small blocks, extracts feature information of each small block of the second sample image, and extracts features from the feature information of each small block to obtain a fourth image feature vector of the second sample image, that is, extracts the overall image feature information of the second sample image through the visual encoder, that is, the fourth image feature vector. The fourth image feature vector can express the image semantic information of the second sample image.

[0164] S803: Input the third text feature vector and the fourth image feature vector into the feature enhancer, and obtain the third fused feature vector of the category label and the second sample image based on the attention extraction mechanism.

[0165] The feature enhancer fuses the third text feature vector and the fourth image feature vector into a third fused feature vector. Optionally, the third text feature vector and the fourth image feature vector are concatenated to obtain a third fused feature vector. The third fused feature vector can express information about the mapping between the category label and the second sample image.

[0166] S804: Input the third fused feature vector into the target decoder to determine the predicted category of the detected object in the second sample image.

[0167] A third fused feature vector obtained by fusing the third text feature vector of the category label and the fourth text feature vector of the second sample image is input into the target decoder, and a predicted category of the detection object in the second sample image is determined based on the target decoder.

[0168] The predicted category of the detection object refers to the category of the detection object in the second sample image predicted by the target decoder, for example, "safety helmet", "escalator", etc.

[0169] S805: Determine a second loss value according to the predicted category and the category label; and train a feature enhancer and a target decoder in the multimodal neural network model according to the second loss value.

[0170] According to the predicted category and the category label of the second sample image, based on the cross entropy loss function, the second loss value is calculated, and according to the second loss value, the parameters of the feature enhancer and the target decoder in the multimodal neural network model are adjusted. Through the input of multiple second sample images and category labels, the feature enhancer and the target decoder in the multimodal neural network model are iteratively trained. When the second loss value reaches the preset target loss value, the feature enhancer and the target decoder in the trained multimodal neural network model are obtained. Or when the number of training times meets the preset target value, the feature enhancer and the target decoder in the trained multimodal neural network model are obtained.

[0171] In an embodiment of the present application, image-text pre-training is performed based on an existing target detection dataset to achieve alignment of visual-text-visual cue features. At this stage, only the feature enhancer and target decoder are trainable, while other components remain frozen. This process is intended to enable the multimodal neural network model to have basic target detection capabilities, for example, the ability to detect basic objects such as "safety helmets" and "seat belts".

[0172] Embodiment 8:

[0173] Fig. 9 A system diagram of a security detection method provided for an embodiment of the present application, wherein specific components include: a text decoder, a visual encoder, a visual cue encoder, a feature enhancer, and a target decoder.

[0174] For example, Fig. 9 As shown, the phrase "a person wearing work clothes and a safety helmet" is input into the text encoder as the text information of the safety detection rule. Optionally, the corresponding object category names in the rule, such as "person", "safety helmet", and "work clothes", can also be input into the text encoder; the collected image of the detection scene is input into the visual encoder; the prompt image of "a person wearing work clothes and a safety helmet" is input into the visual prompt encoder; the feature vectors extracted by the text encoder, the visual encoder, and the visual prompt encoder are fused through the feature enhancer; the fused feature vectors are input into the target decoder, and based on the target decoder, the "person wearing work clothes and a safety helmet" in the collected image of the detection scene is determined.

[0175] Embodiment 9:

[0176] Based on the same application concept, the embodiment of the present application provides a safety detection device, Fig.10 For a schematic diagram of a safety detection device structure provided in an embodiment of the present application, please refer to Fig.10 , the device comprises:

[0177] The acquisition module 1001 is used to receive the text information of the security detection rules inputted from the front-end interface and collect the image to be detected;

[0178] The processing module 1002 is used to input the text information and the image to be detected into a pre-trained multimodal neural network model, and determine the target object in the image to be detected that meets the security detection rule based on the multimodal neural network model; wherein the multimodal neural network model is used to perform feature extraction on the text information to obtain a first text feature vector; perform feature extraction on the image to be detected to obtain a first image feature vector; determine a first fused feature vector based on the first text feature vector and the first image feature vector; and determine the target object in the image to be detected that meets the security detection rule based on the first fused feature vector.

[0179] In one possible implementation, the multimodal neural network model includes: a text encoder, a visual encoder, a feature enhancer, and a target decoder; the processing module 1002 is specifically used to perform feature extraction on the text information based on the text encoder to obtain the first text feature vector; perform feature extraction on the image to be detected based on the visual encoder to obtain the first image feature vector; input the first text feature vector and the first image feature vector into the feature enhancer, and determine the first fused feature vector based on the feature enhancer; input the first fused feature vector into the target decoder, and determine the target object in the image to be detected that meets the security detection rule based on the target decoder.

[0180] In a possible implementation, the processing module 1002 is also used to determine the first sub-text information corresponding to each object in the text information; for each first sub-text information, determine whether the first sub-text information exists in a pre-saved detection object text library; if not, obtain the first image corresponding to the first sub-text information from a pre-saved text image library; input the first image, the text information and the image to be detected into the multimodal neural network model, and determine the target object in the image to be detected that meets the security detection rules based on the multimodal neural network model.

[0181] In a possible implementation, the multimodal neural network model also includes: a visual cue encoder; the processing module 1002 is specifically used to extract features of the first image based on the visual cue encoder to obtain a second image feature vector; the first text feature vector, the first image feature vector and the second image feature vector are input into the feature enhancer, and the first fused feature vector is determined based on the feature enhancer; the first fused feature vector is input into the target decoder, and the target object in the image to be detected that meets the security detection rule is determined based on the target decoder.

[0182] In a possible implementation, the acquisition module 1001 is specifically configured to acquire a first sample image in a first training set and sample text information of a security detection rule corresponding to the first sample image;

[0183] The processing module 1002 is specifically used to input the sample text information into the text encoder to obtain a second text feature vector; input the first sample image into the visual encoder to obtain a third image feature vector; input the second text feature vector and the third image feature vector into the feature enhancer, and determine a second fused feature vector based on the feature enhancer; input the second fused feature vector into the target decoder, and determine the predicted position information of the target object in the first sample image that meets the safety detection rule based on the target decoder; determine a first loss value based on the predicted position information and the labeled position information of the target object in the first sample image that meets the safety detection rule; and train the multimodal neural network model based on the first loss value.

[0184] In a possible implementation, the processing module 1002 is specifically configured to input the first sample image into a pre-trained text description model, and determine sample text information of a security detection rule corresponding to the first sample image based on the text description model.

[0185] In a possible implementation, the acquisition module 1001 is further used to acquire a second sample image and a category label corresponding to the detection object in the second training set;

[0186] The processing module 1002 is also used to input the category label into the text encoder and extract the third text feature vector of the category label; input the second sample image into the visual encoder and extract the fourth image feature vector of the second sample image; input the third text feature vector and the fourth image feature vector into the feature enhancer, and based on the attention extraction mechanism, obtain the third fused feature vector of the category label and the second sample image; input the third fused feature vector into the target decoder to determine the predicted category of the detection object in the second sample image; determine a second loss value based on the predicted category and the category label; and train the feature enhancer and the target decoder in the multimodal neural network model based on the second loss value.

[0187] Embodiment 10:

[0188] Based on the same application concept, an embodiment of the present application provides an electronic device, which can implement the steps of the security detection method discussed above. Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.11As shown, it includes: a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104;

[0189] The memory 1103 stores a computer program. When the program is executed by the processor 1101, the processor 1101 performs the following steps:

[0190] Receive the text information of the security detection rules inputted from the front-end interface and collect the images to be detected;

[0191] Inputting the text information and the image to be detected into a pre-trained multimodal neural network model, and determining the target object in the image to be detected that meets the security detection rule based on the multimodal neural network model;

[0192] Among them, the multimodal neural network model is used to extract features from the text information to obtain a first text feature vector; to extract features from the image to be detected to obtain a first image feature vector; to determine a first fused feature vector based on the first text feature vector and the first image feature vector; and to determine a target object in the image to be detected that meets the security detection rule based on the first fused feature vector.

[0193] In a possible implementation, the multimodal neural network model includes: a text encoder, a visual encoder, a feature enhancer, and a target decoder;

[0194] The process of determining the target object satisfying the safety detection rule in the image to be detected based on the multimodal neural network model includes:

[0195] Extracting features from the text information based on the text encoder to obtain the first text feature vector;

[0196] Extracting features of the image to be detected based on the visual encoder to obtain the first image feature vector;

[0197] Inputting the first text feature vector and the first image feature vector into the feature enhancer, and determining the first fused feature vector based on the feature enhancer;

[0198] The first fused feature vector is input into the target decoder, and a target object satisfying the security detection rule in the image to be detected is determined based on the target decoder.

[0199] In a possible implementation, before determining the target object satisfying the safety detection rule in the image to be detected based on the multimodal neural network model, the method further includes:

[0200] Determine first subtext information corresponding to each object in the text information;

[0201] For each first subtext information, determine whether the first subtext information exists in a pre-stored detection object text library; if not, obtain a first image corresponding to the first subtext information from a pre-stored text image library;

[0202] The process of determining the target object in the image to be detected that satisfies the security detection rule includes:

[0203] The first image, the text information and the image to be detected are input into the multimodal neural network model, and based on the multimodal neural network model, a target object in the image to be detected that meets the security detection rule is determined.

[0204] In a possible implementation, the multimodal neural network model further includes: a visual cue encoder;

[0205] The process of determining the target object in the image to be detected that satisfies the security detection rule includes:

[0206] Extracting features from the first image based on the visual cue encoder to obtain a second image feature vector;

[0207] Inputting the first text feature vector, the first image feature vector and the second image feature vector into the feature enhancer, and determining the first fused feature vector based on the feature enhancer;

[0208] The first fused feature vector is input into the target decoder, and a target object satisfying the security detection rule in the image to be detected is determined based on the target decoder.

[0209] In a possible implementation, the process of pre-training the multimodal neural network model includes:

[0210] Obtaining a first sample image in a first training set and sample text information of a security detection rule corresponding to the first sample image;

[0211] Inputting the sample text information into the text encoder to obtain a second text feature vector; inputting the first sample image into the visual encoder to obtain a third image feature vector;

[0212] Inputting the second text feature vector and the third image feature vector into the feature enhancer, and determining a second fused feature vector based on the feature enhancer;

[0213] Inputting the second fused feature vector into the target decoder, and determining predicted position information of the target object satisfying the safety detection rule in the first sample image based on the target decoder;

[0214] Determine a first loss value based on the predicted position information and the annotated position information of the target object that meets the safety detection rule in the first sample image; and train the multimodal neural network model based on the first loss value.

[0215] In a possible implementation, the process of obtaining the sample text information includes:

[0216] The first sample image is input into a pre-trained text description model, and sample text information of a security detection rule corresponding to the first sample image is determined based on the text description model.

[0217] In a possible implementation manner, before obtaining the first sample image in the first training set and the sample text information of the security detection rule corresponding to the first sample image, the method further includes:

[0218] Obtaining a second sample image and a category label corresponding to the detection object in the second training set;

[0219] Inputting the category label into the text encoder to extract a third text feature vector of the category label; inputting the second sample image into the visual encoder to extract a fourth image feature vector of the second sample image;

[0220] Inputting the third text feature vector and the fourth image feature vector into the feature enhancer, and obtaining a third fused feature vector of the category label and the second sample image based on an attention extraction mechanism;

[0221] Inputting the third fused feature vector into the target decoder to determine the predicted category of the detected object in the second sample image;

[0222] A second loss value is determined according to the predicted category and the category label; and the feature enhancer and the target decoder in the multimodal neural network model are trained according to the second loss value.

[0223] Since the principle of solving the problem by the above electronic device is similar to the method of security detection, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0224] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 1102 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0225] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0226] Embodiment 11:

[0227] Based on the same application concept, the embodiment of the present application provides a computer-readable storage medium, in which a computer program executable by a processor is stored, and when the program is run on the processor, the processor executes any of the security detection methods discussed above. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to the method of security detection, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0228] Embodiment 12:

[0229] Based on the same application concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the security detection methods discussed above. Since the principle of solving the problem by the above computer program product is similar to the security detection method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0230] In an embodiment of the present application, text information of security detection rules inputted by a front-end interface is received, and an image to be detected is collected; the text information and the image to be detected are inputted into a pre-trained multimodal neural network model, and based on the multimodal neural network model, the target object in the image to be detected that meets the security detection rules is determined. Compared with defining rules and training models for each detection scenario, the present application realizes that security detection of different scenarios can be performed by only changing the text information inputted by the front-end interface, thereby realizing security detection of different scenarios using the same model.

[0231] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0232] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0233] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0234] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0235] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for security detection, characterized in that: The method comprises: Receive the text information of the security detection rules inputted from the front-end interface and collect the images to be detected; Inputting the text information and the image to be detected into a pre-trained multimodal neural network model, and determining the target object in the image to be detected that meets the security detection rule based on the multimodal neural network model; Among them, the multimodal neural network model is used to extract features from the text information to obtain a first text feature vector; to extract features from the image to be detected to obtain a first image feature vector; to determine a first fused feature vector based on the first text feature vector and the first image feature vector; and to determine a target object in the image to be detected that meets the security detection rule based on the first fused feature vector.

2. The method according to claim 1, characterized in that The multimodal neural network model includes: a text encoder, a visual encoder, a feature enhancer, and a target decoder; The process of determining the target object satisfying the safety detection rule in the image to be detected based on the multimodal neural network model includes: Extracting features from the text information based on the text encoder to obtain the first text feature vector; Extracting features of the image to be detected based on the visual encoder to obtain the first image feature vector; Inputting the first text feature vector and the first image feature vector into the feature enhancer, and determining the first fused feature vector based on the feature enhancer; The first fused feature vector is input into the target decoder, and a target object satisfying the security detection rule in the image to be detected is determined based on the target decoder.

3. The method according to claim 1, characterized in that Before determining the target object satisfying the safety detection rule in the image to be detected based on the multimodal neural network model, the method further includes: Determine first subtext information corresponding to each object in the text information; For each first subtext information, determine whether the first subtext information exists in a pre-stored detection object text library; if not, obtain a first image corresponding to the first subtext information from a pre-stored text image library; The process of determining the target object in the image to be detected that satisfies the security detection rule includes: The first image, the text information and the image to be detected are input into the multimodal neural network model, and based on the multimodal neural network model, a target object in the image to be detected that meets the security detection rule is determined.

4. The method according to claim 3, characterized in that The multimodal neural network model also includes: a visual cue encoder; The process of determining the target object in the image to be detected that satisfies the security detection rule includes: Extracting features from the first image based on the visual cue encoder to obtain a second image feature vector; Inputting the first text feature vector, the first image feature vector and the second image feature vector into the feature enhancer, and determining the first fused feature vector based on the feature enhancer; The first fused feature vector is input into the target decoder, and a target object satisfying the security detection rule in the image to be detected is determined based on the target decoder.

5. The method according to claim 2, characterized in that: The process of pre-training the multimodal neural network model includes: Obtaining a first sample image in a first training set and sample text information of a security detection rule corresponding to the first sample image; Inputting the sample text information into the text encoder to obtain a second text feature vector; inputting the first sample image into the visual encoder to obtain a third image feature vector; Inputting the second text feature vector and the third image feature vector into the feature enhancer, and determining a second fused feature vector based on the feature enhancer; Inputting the second fused feature vector into the target decoder, and determining predicted position information of the target object satisfying the safety detection rule in the first sample image based on the target decoder; Determine a first loss value based on the predicted position information and the annotated position information of the target object that meets the safety detection rule in the first sample image; and train the multimodal neural network model based on the first loss value.

6. The method according to claim 5, characterized in that The process of obtaining the sample text information includes: The first sample image is input into a pre-trained text description model, and sample text information of a security detection rule corresponding to the first sample image is determined based on the text description model.

7. The method according to claim 5, characterized in that Before obtaining the first sample image in the first training set and the sample text information of the security detection rule corresponding to the first sample image, the method further includes: Obtaining a second sample image and a category label corresponding to the detection object in the second training set; Inputting the category label into the text encoder to extract a third text feature vector of the category label; inputting the second sample image into the visual encoder to extract a fourth image feature vector of the second sample image; Inputting the third text feature vector and the fourth image feature vector into the feature enhancer, and obtaining a third fused feature vector of the category label and the second sample image based on an attention extraction mechanism; Inputting the third fused feature vector into the target decoder to determine the predicted category of the detected object in the second sample image; A second loss value is determined according to the predicted category and the category label; and the feature enhancer and the target decoder in the multimodal neural network model are trained according to the second loss value.

8. A safety detection device, characterized in that: The device comprises: The acquisition module is used to receive the text information of the security detection rules inputted from the front-end interface and collect the image to be detected; A processing module is used to input the text information and the image to be detected into a pre-trained multimodal neural network model, and determine the target object in the image to be detected that meets the security detection rule based on the multimodal neural network model; wherein the multimodal neural network model is used to perform feature extraction on the text information to obtain a first text feature vector; perform feature extraction on the image to be detected to obtain a first image feature vector; determine a first fused feature vector based on the first text feature vector and the first image feature vector; and determine the target object in the image to be detected that meets the security detection rule based on the first fused feature vector.

9. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of the security detection method as claimed in any one of claims 1 to 7 when executing a computer program stored in a memory.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the security detection method as claimed in any one of claims 1 to 7.

Images