A method, apparatus, computer equipment, and readable storage medium for image-based knowledge retrieval to enhance the identification of security risks.

Abstract

The application discloses a method and device for enhancing identification of safety hazards based on picture knowledge retrieval, a computer device and a readable storage medium, which comprises the following steps: firstly, encoding text data in a safety hazard knowledge base to obtain a text feature vector; and then, using a pre-trained feature mapping network to visually encode an image to obtain a local and global feature vector. Target text knowledge is obtained through the vectors for text knowledge recall, a target Prompt template is generated, the image and the template are input into a pre-trained multi-modal large model, and finally, a safety hazard identification result is obtained, so that the accuracy and efficiency of safety hazard identification are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method, apparatus, computer device, and readable storage medium for enhancing knowledge retrieval and identifying security risks based on images. Background Technology

[0002] In the field of safety hazard identification, traditional methods often rely on single-modal data analysis, resulting in low accuracy and poor adaptability to complex scenarios. With the continuous enrichment of image data and related knowledge bases, effectively combining the two has become a research hotspot. However, there are modal differences between visual information and textual knowledge, making direct correlation and utilization difficult. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, computer device, and readable storage medium for enhancing knowledge retrieval and identification of security risks based on images.

[0004] In a first aspect, embodiments of the present invention provide a method for enhancing the identification of security risks based on image-based knowledge retrieval, comprising:

[0005] Text encoding is performed on the text data in the safety hazard knowledge base corresponding to the image to be identified to obtain multiple text feature vectors;

[0006] The image to be identified is visually encoded based on a pre-trained feature mapping network to obtain local feature vectors and global feature vectors corresponding to the image to be identified; the text feature vector, the local feature vector, and the global feature vector are in the same dimension of the feature vector space;

[0007] Based on the local feature vector and the global feature vector, text knowledge retrieval is performed on the multiple text feature vectors to obtain target text knowledge;

[0008] Generate a target Prompt template based on the target text knowledge;

[0009] The image to be identified and the target Prompt template are input into a pre-trained multimodal large model to obtain the security hazard identification result of the image to be identified.

[0010] In one possible implementation, the feature mapping network is acquired through training, including:

[0011] Obtain sample text;

[0012] The masked text is obtained by replacing the keywords in the sample text with preset placeholders, where the keywords are keywords related to security risks.

[0013] The sample text and the masked text are respectively input into the initial text encoder for encoding to obtain the corresponding sample text feature vector and masked text feature vector;

[0014] Based on the sample text feature vector, the masked text feature vector, and Gaussian noise used to bridge the modal gap between the visual feature space and the text feature space, the objective function is: The feature mapping network is trained to obtain the trained feature mapping network;

[0015] in, Let η be the text encoding, η be Gaussian noise, T be the sample text, T' be the masked text, and l be the MSE loss function.

[0016] In one possible implementation, the image to be identified is locally visually encoded based on a pre-trained feature mapping network to obtain a local feature vector corresponding to the image to be identified, including:

[0017] The image to be identified is segmented by average to obtain multiple local images;

[0018] The pre-trained feature mapping network is used to visually encode each of the local images to obtain the local feature vector corresponding to the image to be identified.

[0019] In one possible implementation, the image to be identified is visually encoded based on a pre-trained feature mapping network to obtain a global feature vector corresponding to the image to be identified, including:

[0020] The pre-trained feature mapping network is used to perform global visual encoding on the image to be identified, thereby obtaining the local feature vector corresponding to the image to be identified.

[0021] In one possible implementation, the step of retrieving text knowledge from the plurality of text feature vectors based on the local feature vectors and the global feature vectors to obtain target text knowledge includes:

[0022] Calculate the similarity between each text feature vector and the local feature vector and the global feature vector respectively to obtain the matching score between each text feature vector and the local feature vector and the global feature vector;

[0023] The target text feature vector is determined from the plurality of text feature vectors according to a preset scoring threshold;

[0024] Based on the target text feature vector, text knowledge is retrieved from the safety hazard knowledge base to obtain the target text knowledge.

[0025] In one possible implementation, determining the target text feature vector from the plurality of text feature vectors according to a preset scoring threshold includes:

[0026] The matching score is normalized to obtain the normalized score;

[0027] The normalized score is compared with a preset score threshold, and text feature vectors with scores higher than the score threshold are selected as candidate text feature vectors.

[0028] Cluster analysis is performed based on the semantic relevance of the candidate text feature vectors, and representative text feature vectors are selected from each cluster as the target text feature vectors.

[0029] In one possible implementation, calculating the similarity between each of the text feature vectors and the local feature vectors and the global feature vectors respectively includes:

[0030] Using ColBERT through the formula: Calculate the similarity between each of the text feature vectors and the local feature vectors and the global feature vectors, respectively;

[0031] Where Q = [F M ([φ(r1),φ(r2),...,φ(r9)]),F M (φ(I))],D=F L (d) is the text feature vector, F M ([φ(r1),φ(r2),....,φ(r9)]) is a local feature vector, F M (φ(I)) is the global feature vector.

[0032] Secondly, embodiments of the present invention provide an apparatus for enhancing the identification of security risks based on image-based knowledge retrieval, comprising:

[0033] The encoding module is used to encode the text data in the security hazard knowledge base corresponding to the image to be identified, and obtain multiple text feature vectors; and to perform visual encoding on the image to be identified based on a pre-trained feature mapping network, and obtain the local feature vector and global feature vector corresponding to the image to be identified; the text feature vector, the local feature vector, and the global feature vector are in the same dimension of the feature vector space;

[0034] The recognition module is used to perform text knowledge retrieval on the multiple text feature vectors based on the local feature vectors and the global feature vectors to obtain target text knowledge; generate a target Prompt template based on the target text knowledge; and input the image to be recognized and the target Prompt template into a pre-trained multimodal large model to obtain the security hazard recognition result of the image to be recognized.

[0035] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device performs the method described in the first aspect.

[0036] Fourthly, embodiments of the present invention provide a readable storage medium, the readable storage medium including a computer program, wherein the computer program, when running, controls the computer device where the readable storage medium is located to execute the method described in the first aspect.

[0037] Compared to existing technologies, the beneficial effects of this invention include: employing the method, apparatus, computer equipment, and readable storage medium disclosed herein for image-based knowledge retrieval and enhanced identification of security hazards, the invention encodes textual data in a security hazard knowledge base to obtain textual feature vectors, and then uses a pre-trained feature mapping network to visually encode images to obtain local and global feature vectors. These vectors are used for textual knowledge retrieval to obtain target textual knowledge. After generating a target prompt template, the image and template are input into a pre-trained multimodal large-scale model to finally obtain the security hazard identification result, thus improving the accuracy and efficiency of security hazard identification. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating the steps of a method for enhancing knowledge retrieval and identifying security risks based on images, as provided in an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the cross-framework for enhancing knowledge retrieval and identifying security risks based on images, provided in an embodiment of the present invention.

[0041] Figure 3 A schematic diagram of the training framework of the feature mapping network provided in an embodiment of the present invention;

[0042] Figure 4 A schematic block diagram of the structure of the device for enhancing knowledge retrieval and identifying security risks based on images provided in an embodiment of the present invention;

[0043] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0045] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0046] In order to solve the technical problems mentioned in the background art Figure 1 This is a flowchart illustrating the method for enhancing knowledge retrieval and identifying security risks based on images, as provided in this embodiment. The method for enhancing knowledge retrieval and identifying security risks based on images will be described in detail below.

[0047] Step S201: Text encoding is performed on the text data in the safety hazard knowledge base corresponding to the image to be identified to obtain multiple text feature vectors;

[0048] Step S202: Visually encode the image to be identified based on a pre-trained feature mapping network to obtain the local feature vector and global feature vector corresponding to the image to be identified; the text feature vector, the local feature vector, and the global feature vector are in the same dimension of the feature vector space;

[0049] Step S203: Based on the local feature vector and the global feature vector, perform text knowledge retrieval on the multiple text feature vectors to obtain target text knowledge;

[0050] Step S204: Generate a target Prompt template based on the target text knowledge;

[0051] Step S205: Input the image to be identified and the target Prompt template into a pre-trained multimodal large model to obtain the security hazard identification result of the image to be identified.

[0052] In this embodiment of the invention, for example, assume that the server is processing a task related to identifying safety hazards in industrial plants. The server stores a large knowledge base of safety hazards, containing various textual descriptions related to plant safety, such as "exposed wires in the plant pose a risk of electric shock," "debris piled up in fire exits hinders evacuation," and "no guardrails installed at heights increase the risk of falls," etc. The server first preprocesses this textual data. For example, it may clean the text, removing unnecessary punctuation or special characters, and then perform operations such as stemming or lemmatization to ensure the text is standardized. Next, the server encodes this preprocessed text using a pre-selected text encoder (such as the M3E model). Taking the text "exposed wires in the plant pose a risk of electric shock" as an example, the text encoder will convert it into a text feature vector of a specific dimension (let's say d-dimensional). This feature vector represents the semantic information of the text in a high-dimensional space. For each text in the knowledge base, the server performs the same operation, thereby obtaining multiple text feature vectors. These text feature vectors are like business cards, each representing a specific piece of textual information about security risks in the knowledge base. They all reside in the same high-dimensional feature vector space, facilitating subsequent comparison and correlation calculation with image feature vectors. ## II. The image to be identified is visually encoded based on a pre-trained feature mapping network to obtain the corresponding local and global feature vectors; the text feature vectors, local feature vectors, and global feature vectors reside in the same-dimensional feature vector space.

[0053] The server receives an image of the factory interior to be identified. This image contains many elements, such as various equipment, passageways, and workers. The server first performs an average segmentation process on the image. According to the settings, it is divided into 3x3 image blocks, resulting in 9 local images.

[0054] Taking one local image as an example, this local image could be a corner of a factory, showing some electrical wires and a distribution box. The server uses a pre-trained feature mapping network to visually encode this local image. This feature mapping network, trained on a large amount of previous data, can convert the visual information of this local image into a feature vector. This feature vector is in the same feature vector space as the previous text feature vector. The server performs the same operation on the other eight local images, ultimately obtaining nine local feature vectors, each capturing the detailed features of the corresponding local area in the image. For example, one local feature vector might highlight visual information related to safety hazards, such as the layout and condition of the electrical wires.

[0055] In addition to local encoding of the image, the server also needs to encode the global features of the entire image. For the previously mentioned image of the factory interior, the server uses the same pre-trained feature mapping network to perform global visual encoding of the entire image. During this process, the feature mapping network comprehensively considers all elements in the image, such as the overall layout of the equipment, the direction of the passageways, and the distribution of personnel, and then generates a global feature vector representing the overall semantic information of the entire image. This global feature vector lies in the same feature vector space as the previously obtained local feature vectors and text feature vectors. For example, this global feature vector might reflect macroscopic visual information such as the overall order and safety atmosphere inside the factory.

[0056] The server currently has 9 local feature vectors, 1 global feature vector, and several previously obtained text feature vectors. The server uses the ColBERT algorithm to calculate the similarity between each text feature vector and the local and global feature vectors.

[0057] For example, for the text feature vector corresponding to the text "Exposed wires in the factory pose a risk of electric shock," the server compares it with nine local feature vectors and one global feature vector. Taking one of the local feature vectors (corresponding to a local image containing wires) as an example, the similarity between them is calculated using the ColBERT formula. If the calculated similarity is high, it means that the visual content of this local image is semantically relevant to the text "Exposed wires in the factory pose a risk of electric shock." For other combinations of text feature vectors and local and global feature vectors, the server repeats this calculation process to obtain a matching score for each text feature vector with the local and global feature vectors.

[0058] After obtaining all the matching scores, the server first normalizes these scores. For example, if a text feature vector has matching scores of 0.6 with local feature vectors and 0.7 with global feature vectors, the new scores are obtained after normalization.

[0059] Assuming a preset score threshold of 0.5, the server compares the normalized scores with this threshold. Text feature vectors with scores higher than 0.5 are selected as candidate text feature vectors. For example, after comparison, if 10 text feature vectors have scores higher than the threshold, these become candidate text feature vectors.

[0060] The server performs cluster analysis on these 10 candidate text feature vectors based on semantic relevance. For example, three of the text feature vectors are related to fire safety (such as "debris piled up in fire lanes affects evacuation", "expired fire extinguishers cannot be used normally", "fire hydrants are blocked"), and they may be classified into one category; another four text feature vectors related to electrical safety (such as "exposed wires in the factory pose a risk of electric shock", "distribution box is not grounded", "aging wires", "short circuit in the socket") will be classified into another category, and so on.

[0061] Then, a representative text feature vector is selected from each cluster as the target text feature vector. For example, "Debris piled up in fire lanes affects evacuation" is selected as the representative text feature vector from the fire-related cluster, and "Exposed wires in factory buildings pose a risk of electric shock" is selected as the representative text feature vector from the electrical safety-related cluster.

[0062] The server retrieves textual knowledge from a safety hazard knowledge base based on the determined target text feature vector. For example, for the target text feature vector "exposed wires in the factory pose a risk of electric shock," the server directly retrieves the complete knowledge content of this text from the knowledge base, including possible solutions (such as timely wrapping of wires, replacement of damaged wire insulation, etc.) and relevant knowledge such as the hazard level. The same operation is performed for other target text feature vectors, ultimately obtaining the target text knowledge.

[0063] The server has already obtained the target text knowledge, such as "exposed wires in the factory pose a risk of electric shock" and related solutions. The server will embed this text content into a predefined prompt template.

[0064] Suppose the predefined prompt template is "Please identify the safety hazard in the image based on the following information: [Target text knowledge content], the image is [Image to be identified]". Then, for the target text knowledge "Exposed wires in the factory pose a risk of electric shock", the generated target prompt template could be "Please identify the safety hazard in the image based on the following information: Exposed wires in the factory pose a risk of electric shock, and related solutions include timely wrapping of wires, replacement of damaged wire insulation, etc., the image is [Image of the interior of the factory to be identified]". This target prompt template is designed for effective interaction with multimodal large models.

[0065] The server inputs the image to be identified, which contains the interior scene of the factory, and the target prompt template generated earlier into a pre-trained multimodal large model (such as CLIP or BLIP multimodal models).

[0066] After receiving an image and a Prompt template, the multimodal large model comprehensively considers the visual information in the image and the textual knowledge information in the Prompt template. Taking exposed wires as an example, the model searches for visual features related to "exposed wires" in the image, while combining this with relevant knowledge from the Prompt template (such as electric shock risks and solutions). If exposed wires are indeed found in the image, the model outputs a recognition result containing "There is a safety hazard of exposed wires in the factory," and may also provide some relevant content such as the hazard level and solution suggestions based on the information in the Prompt template. If no safety hazard related to the target textual knowledge is found in the image, the model outputs a result indicating that no relevant safety hazard was found.

[0067] In this embodiment of the invention, the feature mapping network is obtained through the following training, and can be implemented through the following examples.

[0068] Obtain sample text;

[0069] The masked text is obtained by replacing the keywords in the sample text with preset placeholders, where the keywords are keywords related to security risks.

[0070] The sample text and the masked text are respectively input into the initial text encoder for encoding to obtain the corresponding sample text feature vector and masked text feature vector;

[0071] Based on the sample text feature vector, the masked text feature vector, and Gaussian noise used to bridge the modal gap between the visual feature space and the text feature space, the objective function is: The feature mapping network is trained to obtain the trained feature mapping network;

[0072] in, Let η be the text encoding, η be Gaussian noise, T be the sample text, T' be the masked text, and l be the MSE loss function.

[0073] In this embodiment of the invention, for example, the server is responsible for training a feature mapping network, which will be used for image-based security vulnerability identification. First, the server retrieves sample text from a large security-related text database. These sample texts are all related to security vulnerabilities, covering security issues that may arise in various scenarios.

[0074] For example, the sample texts include numerous examples such as "The lack of warning signs at construction sites can easily lead to accidents," "Poor ventilation in chemical workshops may cause toxic gas accumulation," and "Excessive stacking of goods in warehouses poses a risk of collapse." These sample texts were collected from a large number of safety regulations, accident reports, safety inspection records, and other materials. They represent different types of safety hazards and provide a rich source of semantic information for training the feature mapping network.

[0075] The server performs keyword replacement operations on each sample text it receives. Taking the sample text "The lack of warning signs at construction sites can easily lead to accidents" as an example, "construction site" and "warning signs" are keywords closely related to safety hazards.

[0076] The server replaces these keywords with preset placeholders "[$]" according to preset rules, thus transforming the sample text into "[$] not set [$] may easily lead to accidents," which is the masked text. For other sample texts, such as "poor ventilation in a chemical plant may cause the accumulation of toxic gases," the keywords "chemical plant" and "poor ventilation" are replaced to obtain masked text like "[$][$] may cause the accumulation of toxic gases." In this way, the server processes all sample texts to obtain the corresponding masked text. The purpose of this is to allow the model to better understand the structure and semantic relationships of the text during subsequent training, while also helping to reduce the excessive influence of specific keywords on model training, enabling the model to more generalize and handle different safety hazard scenarios.

[0077] The server uses a pre-selected initial text encoder (e.g., the M3E model) to encode the sample text and the masked text. For the sample text "The lack of warning signs at construction sites can easily lead to accidents," the initial text encoder converts it into a sample text feature vector of a specific dimension (let's say d-dimensional). This feature vector represents the semantic information of the sample text in a high-dimensional space, including information about the relationships between concepts such as construction site, warning signs, and accidents.

[0078] Simultaneously, for the corresponding masked text "[$] not set [$] easily leads to accidents", the initial text encoder also converts it into a masked text feature vector of the same dimension d. Although this masked text feature vector loses some specific information due to keyword replacement, it still retains the text structure and some semantic information. For example, it can still reflect the semantic structure that the absence of a certain item at a certain location will lead to accidents. The server performs the same encoding operation on all sample text and masked text, thereby obtaining the corresponding sample text feature vector and masked text feature vector sets.

[0079] During training, the server needs to bridge the modal gap between the visual feature space and the text feature space. To do this, the server adds Gaussian noise to the sample text feature vectors. Through reviewing relevant literature and conducting numerous experiments, the server determined the standard deviation of the Gaussian distribution (for example, setting it to a suitable value).

[0080] Taking the previously obtained feature vector of the sample text "The lack of warning signs at construction sites can easily lead to accidents" as an example, the server adds Gaussian noise to this feature vector according to a selected Gaussian distribution. The feature vector will change to some extent after adding noise, but this change is within a controllable range. The purpose is to make the feature vector more robust and help to narrow the distance from the visual feature space. The server will perform the same operation for other sample text feature vectors.

[0081] The server calculates the loss based on the objective function (MSE loss function). For each pair of sample text feature vectors (after adding Gaussian noise) and masked text feature vectors, the server substitutes them into the objective function for calculation. For example, for a given pair of feature vectors, the MSE loss value between them is calculated.

[0082] The server then uses an optimization algorithm (such as stochastic gradient descent) to minimize this loss function. During training, the model's parameters are continuously adjusted. As training progresses, the gap between the sample text feature vectors (after adding Gaussian noise) and the masked text feature vectors gradually decreases. This means that the feature mapping network can better handle text features and, to some extent, bridges the modal gap between the visual feature space and the text feature space.

[0083] After multiple rounds of training, when the loss function converges to a satisfactory value, the server obtains the trained feature mapping network. This trained feature mapping network can be used for subsequent visual encoding operations on the image to be recognized, thus playing an important role in tasks such as image-based knowledge retrieval and enhanced identification of security risks.

[0084] In this embodiment of the invention, the image to be identified is locally visually encoded based on a pre-trained feature mapping network to obtain the local feature vector corresponding to the image to be identified. This can be implemented through the following example.

[0085] The image to be identified is segmented by average to obtain multiple local images;

[0086] The pre-trained feature mapping network is used to visually encode each of the local images to obtain the local feature vector corresponding to the image to be identified.

[0087] In an embodiment of the present invention, for example, the server receives an image to be identified. This image is an interior scene of a large factory workshop, containing many complex elements, such as various large production equipment, worker operation areas, material storage areas, passages, and electrical lines, which are distributed in different locations throughout the workshop.

[0088] To obtain local features of the image, the server performs average segmentation on the image to be identified according to predetermined rules. Assuming the server is set to segment the image into a 3×3 grid, the image will be uniformly divided into 9 local images.

[0089] Taking one local image as an example, this local image might just cover an electrical control cabinet and a small area around it. In this local image, you can see part of the cabinet door, some indicator lights, some wires connecting the control cabinet, and a small portion of the surrounding ground. Other local images also contain different local scenes within the workshop; for example, some local images show workers operating equipment, while others show a corner where materials are stacked. Through this average segmentation process, the server decomposes the complex overall image into multiple relatively simple sub-images with clear local semantics, providing a foundation for subsequent local visual encoding.

[0090] The server uses a pre-trained feature mapping network to visually encode each segmented local image. Taking the previously mentioned local image containing the electrical control cabinet as an example, the feature mapping network begins processing this local image.

[0091] This feature mapping network, pre-trained on a large amount of diverse image data, is capable of recognizing various visual features in images. When processing this local image, the network captures features such as the shape, color, and texture of the electrical control cabinet, as well as details like the on / off status of indicator lights and the routing of wires. After a series of complex calculations and transformations, the feature mapping network converts the visual information of this local image into a local feature vector of a specific dimension (let's say d-dimensional). This local feature vector acts like a unique identifier for this local image in a high-dimensional feature space, encoding all visual information related to safety hazards in the local image. For example, the appearance of electrical equipment may indicate whether the equipment is operating normally, or whether the routing of wires complies with safety regulations.

[0092] The server performs visual encoding on the remaining eight local images in the same way. For example, for a local image showing a worker operating equipment, the feature mapping network focuses on elements such as the worker's posture, the status of the equipment's interface, and surrounding safety features, converting this visual information into corresponding local feature vectors. For a local image of a corner where materials are stacked, the network captures information such as the stacking height, stacking method, and whether it is near an aisle, generating corresponding local feature vectors.

[0093] After encoding all nine local images, the server obtains nine local feature vectors. These local feature vectors together constitute the set of local feature vectors corresponding to the image to be identified. These local feature vectors, together with the global feature vectors, will be used to subsequently retrieve relevant textual knowledge from the security hazard knowledge base, thereby providing a rich visual information foundation for identifying security hazards in images.

[0094] In this embodiment of the invention, the image to be identified is visually encoded based on a pre-trained feature mapping network to obtain a global feature vector corresponding to the image to be identified. This can be implemented through the following example.

[0095] The pre-trained feature mapping network is used to perform global visual encoding on the image to be identified, thereby obtaining the local feature vector corresponding to the image to be identified.

[0096] In an embodiment of the present invention, for example, the server receives an image of the interior of a large shopping mall to be identified. This image covers a vast space and includes multiple floors, numerous shops, customers, elevators, stairs, fire-fighting facilities, and a complex layout of passageways, among other elements.

[0097] The server begins global visual encoding of the entire image to be identified using a pre-trained feature mapping network. This pre-trained feature mapping network was trained on a large amount of image data from different scenes and types, and it has a strong ability to extract visual features.

[0098] When performing global visual encoding on this image of the mall's interior, the feature mapping network considers all elements in the image simultaneously. For example, it focuses on the mall's overall architectural structure, including the connections between floors, the size and shape of different areas, and so on. For shops, it captures information such as their distribution patterns, shop types (restaurants, clothing, or others), and pedestrian flow at shop entrances.

[0099] Meanwhile, the network doesn't overlook key elements like elevators and stairs. It analyzes elevator operation status (whether it's going up or down, whether the doors are closing properly, etc.), stair width, and pedestrian traffic—all safety-related visual features. Regarding fire safety facilities, it checks the visibility of fire extinguishers and hydrants, and whether they are obstructed. As for customer distribution, it analyzes densely populated and sparsely populated areas, which is crucial for identifying safety hazards (such as the risk of stampedes).

[0100] In addition, channel layout is also a key focus of network monitoring. It determines information such as whether channels are unobstructed, whether there are obstacles blocking them, and whether the channel width meets safety standards.

[0101] After a comprehensive and detailed analysis of the entire image, the feature mapping network integrates all this visual information and, through a series of complex calculations and transformations, ultimately generates a global feature vector of a specific dimension (let's say d-dimensional). This global feature vector is like a comprehensive identifier for this image of the shopping mall's interior in a high-dimensional feature space; it encodes all global visual information related to safety hazards in the entire image.

[0102] For example, this global feature vector might reflect the overall sense of order in a shopping mall. If certain values ​​in the vector indicate a chaotic store layout, narrow passageways, and dense crowds, this could suggest a higher safety risk, such as difficulties in evacuation during a fire or a high risk of stampedes. Conversely, if the vector shows that fire safety facilities are reasonably located and clearly visible, passageways are unobstructed, and people are evenly distributed, then the overall safety situation is likely to be relatively good.

[0103] This global feature vector, together with the previously obtained local feature vectors, will be used to retrieve relevant textual knowledge from the safety hazard knowledge base, thus providing a comprehensive visual information foundation for accurately identifying safety hazards in this large shopping mall's interior image. In this way, both local, detailed safety hazards (such as electrical problems in a shop, which can be reflected by local feature vectors) and macro-level safety hazards affecting the entire mall (such as overall evacuation capacity, which can be reflected by global feature vectors) can be effectively identified.

[0104] In this embodiment of the invention, the step of retrieving text knowledge from the plurality of text feature vectors based on the local feature vectors and the global feature vectors to obtain target text knowledge can be implemented through the following example.

[0105] Calculate the similarity between each text feature vector and the local feature vector and the global feature vector respectively to obtain the matching score between each text feature vector and the local feature vector and the global feature vector;

[0106] The target text feature vector is determined from the plurality of text feature vectors according to a preset scoring threshold;

[0107] Based on the target text feature vector, text knowledge is retrieved from the safety hazard knowledge base to obtain the target text knowledge.

[0108] In this embodiment of the invention, for example, the server has already obtained the local feature vector and global feature vector corresponding to the image to be identified (e.g., an image of the interior of a school building), as well as multiple text feature vectors from a security hazard knowledge base. Now, similarity is calculated to obtain a matching score.

[0109] Taking a text feature vector from the knowledge base as an example, the text corresponding to this text feature vector is "The lighting facilities in the stairwell of the teaching building are damaged, posing a risk of falling." The server uses the ColBERT algorithm to calculate the similarity between this text feature vector and the local feature vectors of the image.

[0110] Suppose the image is segmented into 9 local images, each with a corresponding local feature vector. For one of these local feature vectors, such as the local feature vector corresponding to the stairwell image, the server calculates it using the ColBERT algorithm as follows:

[0111] First, the algorithm extracts semantic information related to the stairwell from the text feature vector, such as the representation of concepts like "stairwell," "lighting facilities," and "fall risk" in a high-dimensional vector space. Simultaneously, it extracts visual feature representations of the stairwell from the local feature vectors, such as the shape and color of the stairs, and the surrounding environment. Then, according to ColBERT's similarity calculation formula, it compares and calculates this semantic and visual information to obtain a preliminary similarity score, for example, 0.6.

[0112] This calculation process is repeated for the other 8 local feature vectors to obtain the similarity scores of 9 local feature vectors related to this text feature vector.

[0113] Next, the server calculates the similarity between this text feature vector and the global feature vector of the image. For images of the interior of a school building, the global feature vector contains information such as the overall layout of the building, the distribution of various areas, and the overall flow of people.

[0114] The server also uses the ColBERT algorithm to extract semantic information related to the overall safety hazards of the teaching building from the text feature vector, and to extract the overall visual feature representation of the teaching building from the global feature vector. Then, a similarity score is calculated, assuming the obtained score is 0.5.

[0115] To obtain the matching score between this text feature vector and both the local and global feature vectors, the server employs a comprehensive strategy. For example, it may take the larger of the maximum similarity score with the local feature vector (here, 0.6) and the similarity score with the global feature vector (0.5), i.e., 0.6, as the matching score for this text feature vector.

[0116] The server will repeat the above calculation process for each text feature vector in the knowledge base to obtain the matching degree score between each text feature vector and the local and global feature vectors.

[0117] After obtaining the matching scores of all text feature vectors, the server first performs normalization. For example, the matching score may range from 0 to 1. The original matching score of a certain text feature vector is 0.6, and after normalization, it may become 0.8 (assuming that the normalization method is to divide the original score by the maximum value among all scores).

[0118] Assume the preset score threshold is 0.7. The server compares the normalized scores with this threshold. Text feature vectors with scores higher than 0.7 are selected as candidate text feature vectors. For example, after comparison, if five text feature vectors have scores higher than the threshold, these become candidate text feature vectors.

[0119] The server performs clustering analysis on these 5 candidate text feature vectors based on semantic relevance. For example, two of the text feature vectors are related to fire protection facilities (such as "fire hydrants in the teaching building are dry and cannot extinguish fires" and "fire extinguishers are expired and cannot be used properly"), and they may be classified into one category; the other 3 text feature vectors related to stair safety (such as "lighting facilities in the stairwell of the teaching building are damaged and there is a risk of falling" and "stair handrails are loose and there is a risk of falling" and "stair steps are broken and easy to trip over") will be classified into another category.

[0120] Then, a representative text feature vector is selected from each cluster as the target text feature vector. For example, "fire hydrants in the teaching building are dry and cannot be used to extinguish fires" is selected as the representative text feature vector from the cluster related to fire protection facilities, and "lighting facilities in the stairwell of the teaching building are damaged and there is a risk of falling" is selected as the representative text feature vector from the cluster related to stair safety.

[0121] The server retrieves textual knowledge from the safety hazard knowledge base based on the determined target text feature vector. For the target text feature vector "fire hydrants in the teaching building are dry and cannot be used to extinguish fires," the server searches for the corresponding complete textual knowledge in the safety hazard knowledge base. This complete textual knowledge can include detailed reasons why the fire hydrants are dry (such as long-term lack of inspection and maintenance, pipe rupture, etc.), the possible serious consequences (such as the fire not being extinguished in time, leading to the spread of the fire, endangering the lives of teachers and students and the safety of the teaching building), and corresponding solutions (such as immediately inspecting and repairing the pipes, refilling with water, etc.).

[0122] For another target text feature vector, “Damaged lighting facilities in the stairwell of the teaching building pose a risk of falling,” the server also retrieves its complete text knowledge from the knowledge base, including common causes of lighting facility damage (such as broken light bulbs, circuit failures, etc.), specific assessments of the risk of falling (such as assessing the risk level based on factors such as the flow of people in the stairwell and the slope of the stairs), and solutions (such as timely replacement of light bulbs, repair of circuits, etc.).

[0123] Through this text knowledge retrieval, the server obtains the target text knowledge, which will be used to generate the target Prompt template, thereby providing important text information for accurately identifying safety hazards in images inside the teaching building.

[0124] In this embodiment of the invention, the step of determining the target text feature vector from the plurality of text feature vectors according to a preset scoring threshold can be implemented through the following example.

[0125] The matching score is normalized to obtain the normalized score;

[0126] The normalized score is compared with a preset score threshold, and text feature vectors with scores higher than the score threshold are selected as candidate text feature vectors.

[0127] Cluster analysis is performed based on the semantic relevance of the candidate text feature vectors, and representative text feature vectors are selected from each cluster as the target text feature vectors.

[0128] In this embodiment of the invention, for example, the server has calculated the matching score between each text feature vector and the local and global feature vectors. Suppose the server is processing a task related to identifying safety hazards in a hospital, and has obtained numerous text feature vectors from a safety hazard knowledge base. The corresponding text content includes phrases such as "A malfunctioning call system in the ward affects patients' ability to call for help," "Inadequate disinfection in the operating room easily leads to infection," and "Debris piled up in the corridor obstructs stretcher passage," etc. The server has also calculated the matching score of these text feature vectors for a single image of the hospital interior to be identified.

[0129] Taking the text feature vector corresponding to "The ward call system malfunction affects patients' ability to call for help" as an example, its matching score is 0.6, while the maximum matching score among all text feature vectors is 0.8. The server uses a simple normalization method, which divides the matching score of each text feature vector by the maximum value. Therefore, the normalized score of this text feature vector is 0.6 / 0.8 = 0.75.

[0130] For the text feature vector corresponding to "inadequate disinfection in the operating room can easily lead to infection," assuming its matching score is 0.5, the normalized score would be 0.5 / 0.8 = 0.625. The server will normalize all calculated matching scores in this way to obtain the normalized score for each text feature vector. The purpose of this is to unify all scores to the same scale, facilitating subsequent comparisons with a preset score threshold.

[0131] The server pre-sets a scoring threshold, let's say it's 0.7. After normalization, the server starts comparing the normalized score of each text feature vector with this threshold.

[0132] For the text feature vector corresponding to "The ward call system failure affects the patient's call for help" which has a normalized score of 0.75, since 0.75 > 0.7, this text feature vector is selected as a candidate text feature vector.

[0133] However, the text feature vector corresponding to "inadequate disinfection in the operating room can easily lead to infection" with a normalized score of 0.625 will not be selected because 0.625 < 0.7.

[0134] The server iterates through all normalized text feature vectors, selecting those with a score higher than 0.7 as candidate text feature vectors. For example, another text feature vector corresponding to the text "unclear parking space markings in hospital parking lots lead to parking chaos" has a normalized score of 0.72, and this text feature vector will also be selected as a candidate text feature vector. Through this comparative selection, the server identifies a subset of candidate text feature vectors that have a high visual correlation with the image to be recognized from among numerous text feature vectors. These candidate text feature vectors will serve as the basis for further analysis.

[0135] The server now performs semantic relevance clustering analysis on the selected candidate text feature vectors. Suppose that after the previous steps, 5 candidate text feature vectors were selected, and the corresponding texts are "The call system failure in the ward affects patients' ability to call for help", "The oxygen supply equipment failure in the ward endangers the patient's life", "The unclear parking space markings in the hospital parking lot lead to parking chaos", "The emergency call button in the hospital elevator car is malfunctioning", and "The disorderly placement of medicines in the hospital pharmacy affects the efficiency of medicine dispensing".

[0136] By analyzing the semantic content of these texts, the server found that "the malfunction of the call system in the ward affects patients' ability to call for help" and "the malfunction of the oxygen supply equipment in the ward endangers patients' lives" are both related to the malfunction of medical equipment in the ward. The feature vectors of these two texts have a high semantic correlation, so they are clustered into one category.

[0137] The phrase "unclear parking space markings in the hospital parking lot lead to parking chaos" differs significantly in semantics from the other texts and thus forms its own category. While "malfunctioning emergency call buttons in hospital elevators" differs from equipment malfunctions in patient wards, both fall under the category of hospital facility malfunctions. Therefore, the feature vectors of the two texts related to patient wards are somewhat correlated within a broad semantic category and are also included in the cluster of patient ward equipment malfunctions. The phrase "disorganized drug placement in the hospital pharmacy affects medication dispensing efficiency" forms its own category because it primarily concerns pharmacy management issues.

[0138] Representative text feature vectors are selected from each cluster as target text feature vectors. For the cluster of "ward equipment failure," the server selects "ward call system failure affects patient emergency calls" as the representative text feature vector. This is because this text typically reflects the impact of ward equipment failure on patient safety within this cluster.

[0139] For parking lot-related clusters, the unique text feature vector "unclear parking space markings in hospital parking lots lead to parking chaos" is directly used as the representative text feature vector of this cluster.

[0140] For clustering in pharmacy management, "disorganized drug placement in hospital pharmacies affects dispensing efficiency" is used as a representative text feature vector. In this way, the server determines the target text feature vector from the candidate text feature vectors. These target text feature vectors can more accurately represent the safety hazard knowledge related to the image to be identified, providing crucial textual information for subsequent safety hazard identification.

[0141] In this embodiment of the invention, the step of calculating the similarity between each of the text feature vectors and the local feature vectors and the global feature vectors can be performed through the following example.

[0142] Using ColBERT through the formula: Calculate the similarity between each of the text feature vectors and the local feature vectors and the global feature vectors, respectively;

[0143] Where Q = [F M ([φ(r1),φ(r2),...,φ(r9)]),F M (φ(I))],D=F L (d) is the text feature vector, F M ([φ(r1),φ(r2),....,φ(r9)]) is a local feature vector, F M (φ(I)) is the global feature vector.

[0144] In this embodiment of the invention, for example, it is assumed that the server is processing a task related to identifying security risks in an airport terminal. The server has already obtained multiple text feature vectors from a security risk knowledge base, and has also processed the airport terminal image to be identified to obtain local feature vectors.

[0145] Taking a text feature vector as an example, the text corresponding to this text feature vector is "Exposed wires near seats in the waiting hall pose a risk of electric shock". This text feature vector represents semantic information about exposed wires near seats in the waiting hall in a high-dimensional space.

[0146] Meanwhile, assuming the image is segmented into multiple local images, one of which has a local feature vector that encompasses visual information about a seating area in the waiting hall, including the seating layout, surrounding facilities, and floor conditions.

[0147] When calculating similarity, ColBERT first preprocesses the text feature vector and local feature vectors. For the text feature vector, it analyzes and extracts the representations of each semantic element (such as "waiting hall," "seat," "exposed wire," and "electric shock risk") in high-dimensional space from the sentence "There is a risk of electric shock due to exposed wires near seats in the waiting hall." For the local feature vector, it extracts the representations of visual elements related to the seating area (such as the shape and color of the seats, and the outlines of nearby objects) in the feature vector.

[0148] Then, according to the formula, the server uses ColBERT to substitute these representations into the formula for calculation. For example, for the semantic element "exposed wires" in the text feature vector, ColBERT will find the visual element representation in the local feature vector that best matches it, which could be some lines resembling wire shapes or partial visual features of electrical appliances. Through matching and calculating these elements, a similarity score is finally obtained. Suppose this similarity score is 0.6, it means that this text feature vector and this local feature vector have a certain degree of correlation.

[0149] The server will repeat the above calculation process for this text feature vector and all local feature vectors. For example, another local feature vector corresponds to a local image of the area near the boarding gate in the waiting hall. This local feature vector contains visual information such as boarding gate facilities, signs, and passenger queues.

[0150] ColBERT then analyzes and calculates the text feature vector and this local feature vector again according to the above calculation logic. Since this local feature vector has little semantic correlation with the text "exposed wires near the seats in the waiting hall", the calculated similarity score may be low, assumed to be 0.1.

[0151] The server will calculate the similarity score between each text feature vector in the security vulnerability knowledge base and each local feature vector of the image in the above manner, thereby obtaining the similarity score between each text feature vector and all local feature vectors.

[0152] Let's continue using the text feature vector "Exposed wires near seats in the waiting hall pose a risk of electric shock" as an example. Now we need to calculate its similarity to the global feature vector of the airport terminal image.

[0153] The global feature vector of an airport terminal includes comprehensive visual information such as the overall architectural structure of the terminal (e.g., the shape of the hall, number of floors, and layout of passageways), the overall distribution of people (e.g., areas where passengers gather and the distribution of staff), and the overall layout of major facilities (e.g., the distribution of boarding gates, security checkpoints, and shops).

[0154] ColBERT also preprocesses the text feature vector and the global feature vector. For the text feature vector, it analyzes and extracts the representations of semantic elements such as "waiting hall," "seats," "exposed wires," and "electric shock risk" in high-dimensional space. For the global feature vector, it extracts the representations of visual elements related to the overall layout and safety of the terminal (such as the overall electrical facility layout of the hall, the relationship between densely populated areas and electrical equipment, etc.) in the feature vector.

[0155] Then, calculations are performed according to the formula. During this calculation, since the global feature vector encompasses information from the entire terminal, ColBERT needs to find matches between textual semantic elements and visual elements at a more macroscopic level. For example, although the text is about exposed wires near seats, from a global perspective, it's necessary to consider the relationship between the entire terminal's electrical system layout and the seating area, as well as the impact of passenger flow on the risk of electric shock. After complex calculations, assuming a similarity score of 0.5 is obtained, it indicates that this textual feature vector has a certain degree of correlation with the global feature vector.

[0156] The server repeats the process of calculating the similarity between each text feature vector and the global feature vector for every text feature vector in the security hazard knowledge base. For example, if another text feature vector corresponds to the text "Airport terminal fire escape is blocked, affecting evacuation," ColBERT will use the same calculation logic to preprocess this text feature vector and the global feature vector, and then calculate their similarity according to the formula. In this way, the server obtains the similarity score between each text feature vector and the global feature vector.

[0157] Through the detailed calculation process described above, the server uses ColBERT to calculate the similarity between each text feature vector and the local and global feature vectors according to the given formula, providing an important data foundation for subsequent text knowledge retrieval and security risk identification.

[0158] To more clearly describe the solutions provided in the embodiments of the present invention, a more detailed implementation method is provided below.

[0159] (I) Knowledge Recall Based on Visual Feature Vectors

[0160] This invention proposes a text knowledge retrieval method based on visual feature vectors, which is particularly suitable for extracting relevant text knowledge from image data to assist in the application of multimodal models for detecting image security risks. Please refer to the following: Figure 2 The specific implementation steps are as follows:

[0161] Text Feature Vector Generation: First, the text data in the knowledge base is preprocessed and encoded. Then, a text encoder is applied to convert each piece of text data into a corresponding feature vector D = F. L (d), where d represents the content of the text. This step transforms the original text into points in a high-dimensional vector space, facilitating subsequent similarity calculations.

[0162] Image segmentation and local feature vector generation: For the image to be recognized, it is first segmented into 3*3 image blocks, resulting in 9 independent image regions. Next, each of these 9 image blocks is visually encoded, and a mapping network is used to generate a feature vector F corresponding to each image block. M ([φ(r1),φ(r2),....,φ(r9)]), where r i Let represent the i-th image patch, and φ represent the image encoding. This process aims to capture detailed features of local regions of the image, providing a more refined feature description for subsequent recall tasks.

[0163] Global Feature Vector Generation: In addition to extracting local feature vectors, global features of the entire image are encoded. A visual encoder and a mapping network are used to generate the global feature vector F of the original image.M (φ(I)), where I represents the entire image and φ represents the image encoding. This step ensures a grasp of the overall semantic information of the image, providing a global perspective for comprehensive analysis.

[0164] Textual Knowledge Retrieval: To retrieve image-related textual knowledge from a knowledge base, this paper proposes using ColBERT to compute the image feature vector Q = [F...]. M ([φ(r1),φ(r2),...,φ(r9)]),F M (φ(I))] and the text feature vector D=F L The similarity between (d) and (d). The specific formula for calculating the similarity is:

[0165] Specifically, by comparing the local and global feature vectors of the image with the feature vectors of the text, the maximum matching score between them is calculated to measure the relevance between the image and the text. The higher this similarity score, the stronger the relevance between the text and the image, and thus the more accurately relevant text knowledge [T1,T2] can be retrieved from the knowledge base.

[0166] Prompt Template Generation: Once image-related textual knowledge is retrieved, this textual content is embedded into a predefined prompt template. This template is designed for further interaction with the multimodal large model. Combining the original image as model input and the generated prompt template, the system can ultimately identify potential security risks or other information in the image.

[0167] (ii) Using only text training mapping networks

[0168] This invention provides an improved training method for Visual-Language Models (VLMs), which significantly enhances the performance of multimodal tasks by optimizing the collaboration between image encoders and text encoders. Specifically, this invention utilizes a relatively mature visual encoder already trained in the prior art to encode images, while simultaneously employing an equally mature text encoder to encode text, thereby effectively reducing the complexity of the training process and the computational resources required.

[0169] Traditional multimodal learning models, such as CLIP, rely on large-scale image data for contrastive learning. However, in practical applications, insufficient image data often exists, making model training through contrastive learning difficult. To address this issue, this invention proposes a mapping network strategy that trains solely on text, thus bypassing the dependence on large amounts of image data. The core of this strategy lies in replacing traditional image-text joint training with training a text feature mapping network, while employing a method of fusing Gaussian noise to bridge the modal gap between the visual feature space and the text feature space.

[0170] In the specific implementation of this invention, a pre-trained visual-language model (such as CLIP or BLIP) is first used to map the input image and text into feature vector spaces of the same dimension. However, due to the inherent feature space differences between the visual modality and the text modality, i.e., the visual feature space and the text feature space are not perfectly aligned, this modality gap will adversely affect the final performance of the model, thereby limiting the application effect of visual feature vectors.

[0171] To effectively address the aforementioned problems, this invention proposes a scheme to reduce modal gaps by injecting noise or employing a projection-based alignment method. Specifically, this invention employs a noise addition strategy, namely, injecting Gaussian noise into the feature vector space to mitigate the modal gap between visual and text features. Furthermore, to further optimize this alignment process, this invention studies the distribution of Gaussian noise and identifies a comparable Gaussian noise distribution function, thereby achieving significant improvements in the alignment of visual and text features.

[0172] Building upon this foundation, this invention overcomes the training difficulties caused by insufficient image data by employing a method that trains solely on text, significantly improving the overall performance of multimodal tasks. This innovative training method enables efficient and accurate visual-language model training with limited computational resources, even under conditions of limited data. Furthermore, this method has broad applicability, achieving excellent performance across a variety of multimodal tasks.

[0173] In summary, this invention overcomes the shortcomings of traditional methods by optimizing the collaboration between image and text encoders, using Gaussian noise to reduce modal gaps, and training only on text, and provides an efficient and reliable new approach for training visual-language models.

[0174] (III) Mapping Network Training Process

[0175] This invention provides a specific implementation method for training a visual-language model (VLM), which optimizes model performance by introducing text masking and Gaussian noise. Combined with... Figure 3 The following is a detailed description of the implementation process, as demonstrated in the presentation:

[0176] First, in text x c In the preprocessing stage, all "keywords" in the text to be processed are replaced with predefined placeholders "[$]". For example, for the text "safety exit of the factory", its keywords include "factory" and "safety exit", so the text will be converted to "[$] of [$]". This converted text is called the masked text x. t .

[0177] During the encoding step, the original text and the masked text are input into the same text encoder to generate corresponding feature vectors. Let's say the original text is x. c Feature vector e generated by the text encoder c Set as mask text x t Feature vector e generated by the text encoder t The text encoder uses a well-trained model, such as M3E, which can effectively capture the semantic features of the text.

[0178] To further narrow the gap between visual and textual modalities, this invention introduces a Gaussian noise-based strategy for modal alignment. Specifically, Gaussian noise η is added to the text feature vectors, making the feature vectors more robust and helping to bridge the modal gap between the visual and textual feature spaces. The standard deviation of this Gaussian distribution is set to ε. Through reviewing relevant literature and conducting extensive experiments, this invention selects a standard deviation value of ε. Gaussian noise η is used to ensure that the main features of the text are preserved during modal alignment, while effectively reducing the negative impact of modal gaps on model performance.

[0179] Next, the training objective of this invention is to minimize the following loss function:

[0180] in Let η represent text encoding, η be Gaussian noise, T be the text, T' be the masked text, and l be the MSE loss function;

[0181] By minimizing the above loss function, this invention effectively trains the mapping network F. MThis approach aligns textual and visual features more closely, thereby improving the overall performance of multimodal tasks. In particular, because training is performed only on the text portion, this method demonstrates good performance even with limited data resources, effectively reducing computational resource consumption.

[0182] The training method based on text masking and Gaussian noise proposed in this invention significantly improves the performance of visual-language models and has high practical application value and broad application prospects.

[0183] This invention offers several significant advantages. Its core lies in accurately recalling image-related textual knowledge by combining local and global image features, thereby effectively supporting the execution of multimodal tasks. Compared to traditional methods, this invention requires no additional textual data, significantly reducing model complexity and making the entire application process simpler and easier. Furthermore, it greatly simplifies the human interaction process in practical applications; this non-interactive recognition method enhances the system's automation level.

[0184] Traditional multimodal alignment typically requires large amounts of image and text data for training to ensure accurate mapping between visual and linguistic features. However, acquiring and processing large-scale multimodal data is not only time-consuming and labor-intensive but also places extremely high demands on computational resources. To address this challenge, this invention proposes a method that uses only text for training. Specifically, it optimizes the text feature mapping network to achieve effective alignment between visual and text features. This eliminates the reliance on massive image data for model training, overcoming the training difficulties caused by insufficient image data.

[0185] Please refer to the following: Figure 4 , Figure 4 An apparatus 110 for enhancing knowledge retrieval and identifying security risks based on images, provided in an embodiment of the present invention, includes:

[0186] Encoding module 1101 is used to encode text data in a knowledge base of security risks corresponding to the image to be identified, and obtain multiple text feature vectors; and to perform visual encoding on the image to be identified based on a pre-trained feature mapping network, and obtain local feature vectors and global feature vectors corresponding to the image to be identified; the text feature vectors, the local feature vectors and the global feature vectors are in the same feature vector space;

[0187] The recognition module 1102 is used to perform text knowledge retrieval on the multiple text feature vectors based on the local feature vectors and the global feature vectors to obtain target text knowledge; generate a target Prompt template based on the target text knowledge; and input the image to be recognized and the target Prompt template into a pre-trained multimodal large model to obtain the security hazard recognition result of the image to be recognized.

[0188] It should be noted that the implementation principle of the aforementioned image-based knowledge retrieval enhancement security hazard identification device 110 can refer to the implementation principle of the aforementioned image-based knowledge retrieval enhancement security hazard identification method, and will not be repeated here. It should be understood that the division of the various modules in the above device is merely a logical functional division; in actual implementation, they can be fully or partially integrated into a single physical entity, or physically separated. Furthermore, these modules can all be implemented in software through processing element calls; they can all be implemented in hardware; or some modules can be implemented by processing element calls to software, and some modules by hardware. For example, the image-based knowledge retrieval enhancement security hazard identification device 110 can be a separately established processing element, or it can be integrated into a chip in the aforementioned device. Alternatively, it can be stored as program code in the memory of the aforementioned device, and called and executed by a processing element of the aforementioned device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step or module of the above method can be completed by the integrated logic circuit in the hardware of the processor element or by instructions in the form of software.

[0189] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0190] This invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned device 110 for enhancing knowledge retrieval and identifying security vulnerabilities based on images. Figure 5 As shown, Figure 5 This is a structural block diagram of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a device 110 for enhancing the identification of security risks through image-based knowledge retrieval, a memory 111, a processor 112, and a communication unit 113.

[0191] To enable data transmission or interaction, the memory 111, processor 112, and communication unit 113 are electrically connected to each other directly or indirectly. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The image-based knowledge retrieval enhancement and security vulnerability identification device 110 includes at least one software function module that can be stored in the memory 111 or embedded in the operating system (OS) of the computer device 100 in the form of software or firmware. The processor 112 is used to execute the image-based knowledge retrieval enhancement and security vulnerability identification device 110 stored in the memory 111, such as the software function module and computer program included in the image-based knowledge retrieval enhancement and security vulnerability identification device 110.

[0192] This invention provides a readable storage medium, which includes a computer program. When the computer program runs, it controls the computer device where the readable storage medium is located to execute the aforementioned device 110 for enhancing knowledge retrieval and identifying security risks based on images.

[0193] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the foregoing illustrative discussions are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in accordance with the foregoing teachings. These embodiments were chosen and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the disclosure and to employ various embodiments with different modifications to suit a particular intended application.

Claims

1. A method for enhancing the identification of safety hazards by knowledge retrieval based on pictures, characterized in that, The method comprises the following steps: text encoding is performed on text data in a safety hazard knowledge base corresponding to a to-be-identified image to obtain a plurality of text feature vectors; visual encoding is performed on the to-be-identified image based on a pre-trained feature mapping network to obtain a local feature vector and a global feature vector corresponding to the to-be-identified image; the text feature vector, the local feature vector and the global feature vector are in the same dimension feature vector space; text knowledge recall is performed on the plurality of text feature vectors according to the local feature vector and the global feature vector to obtain target text knowledge; a target Prompt template is generated based on the target text knowledge; the to-be-identified image and the target Prompt template are input into a pre-trained multi-modal large model to obtain a safety hazard identification result of the to-be-identified image; the feature mapping network is obtained by the following training, comprising: obtaining sample text; replacing a keyword in the sample text with a preset placeholder to obtain a masked text, the keyword being a keyword related to a safety hazard; respectively inputting the sample text and the masked text into an initial text encoder for encoding to obtain corresponding sample text feature vectors and masked text feature vectors; Based on the sample text feature vector, the mask text feature vector and the Gaussian noise for bridging the inter-modal gap between the visual feature space and the text feature space, the target function is: The mapping network function to be trained is trained to obtain the trained feature mapping network. wherein, for text encoding, η Gaussian noise, T is the sample text, T' is the masked text, λ is the MSE loss function, Γ represents the training text dataset, F M a mapping network function to be trained; the text knowledge recall performed on the plurality of text feature vectors according to the local feature vector and the global feature vector to obtain target text knowledge comprises: respectively calculating the similarity of each text feature vector with the local feature vector and the global feature vector to obtain a matching degree score of each text feature vector with the local feature vector and the global feature vector; determining a target text feature vector from the plurality of text feature vectors according to a preset score threshold; performing text knowledge recall from the safety hazard knowledge base based on the target text feature vector to obtain target text knowledge.

2. The method of claim 1, wherein, the local visual encoding performed on the to-be-identified image based on the pre-trained feature mapping network to obtain the local feature vector corresponding to the to-be-identified image comprises: performing average segmentation processing on the to-be-identified image to obtain a plurality of local images; respectively performing visual encoding on each local image by using the pre-trained feature mapping network to obtain the local feature vector corresponding to the to-be-identified image.

3. The method of claim 1, wherein, the global visual encoding performed on the to-be-identified image by using the pre-trained feature mapping network to obtain the global feature vector corresponding to the to-be-identified image. the determination of the target text feature vector from the plurality of text feature vectors according to the preset score threshold comprises:

4. The method of claim 1, wherein, performing normalization processing on the matching degree score to obtain a normalized score; comparing the normalized score with a preset score threshold to screen out a text feature vector with a score higher than the score threshold as a candidate text feature vector; performing clustering analysis according to the semantic correlation of the candidate text feature vector to select a representative text feature vector from each cluster as the target text feature vector. ​ 5. The method of claim 1, wherein, The calculating the similarity of each of the text feature vectors and the local feature vector and the global feature vector respectively comprises: By using ColBERT, the similarity between each of the text feature vectors and the local feature vector and the global feature vector is calculated respectively by the formula: By using ColBERT, the similarity between each of the text feature vectors and the local feature vector and the global feature vector is calculated respectively by the formula: By using ColBERT, the similarity between each of the text feature vectors and the local feature vector and the global wherein, l D denotes the total number of text knowledge blocks, Q = [F M ([φ(r1),φ(r2),...,φ(r9)]),F M (φ(I))], D = F L (d) is a text feature vector, F M ([φ(r1),φ(r2),....,φ(r9)]) is a local feature vector, F M (φ(I)) is a global feature vector, Q denotes an image feature vector, F M is a mapping network function to be trained, φ() denotes an image encoder, r1-r9 denote subgraphs of an image, the image is divided into 9 blocks, each block is r i , I denotes an entire image, D denotes a text block feature vector, F L () denotes a text encoding, is a cross-modal similarity function, d denotes a text knowledge block, denotes an image query vector.

6. A device for enhancing the identification of security risks through image-based knowledge retrieval, characterized in that, The method comprises: The encoding module is configured to perform text encoding on text data in a safety hazard knowledge base corresponding to the to-be-identified image to obtain a plurality of text feature vectors, and perform visual encoding on the to-be-identified image based on a pre-trained feature mapping network to obtain a local feature vector and a global feature vector corresponding to the to-be-identified image; the text feature vectors, the local feature vector, and the global feature vector are in a same-dimensional feature vector space; The identification module is configured to perform text knowledge recall on the plurality of text feature vectors based on the local feature vector and the global feature vector to obtain target text knowledge, and generate a target Prompt template based on the target text knowledge. The to-be-identified image and the target Prompt template are input into a pre-trained multi-modal large model to obtain a safety hazard identification result of the to-be-identified image. The feature mapping network is obtained through the following training and comprises: Obtaining a sample text; replacing a keyword in the sample text with a preset placeholder to obtain a masked text, the keyword being a keyword related to a safety hazard; inputting the sample text and the masked text into an initial text encoder respectively for encoding to obtain corresponding sample text feature vectors and masked text feature vectors; based on the sample text feature vectors, the masked text feature vectors and Gaussian noise for bridging a modal gap between a visual feature space and a text feature space, obtaining a feature mapping network through a target function: Training the mapping network function to be trained to obtain a trained feature mapping network; wherein, is a text encoding, η is Gaussian noise, T is a sample text, T' is a masked text, λ is an MSE loss function, Γ represents a training text data set, F M the mapping network function to be trained; The identification module is specifically configured to: The calculating the similarity of each of the text feature vectors and the local feature vector and the global feature vector respectively comprises:

7. A computer device, comprising: The computer device comprises a processor and a non-volatile memory having computer instructions stored therein, and when the computer instructions are executed by the processor, the computer device executes the method according to any one of claims 1-5.

8. A readable storage medium, characterized by, The readable storage medium comprises a computer program, and when the computer program runs, controls a computer device where the readable storage medium is located to execute the method according to any one of claims 1-5.

Images