Method and device for performing knowledge retrieval based on picture to enhance identification of potential safety hazard, computer equipment and readable storage medium

Abstract

The invention discloses a method and a device for performing knowledge retrieval based on pictures to enhance recognition of potential safety hazards, computer equipment and a readable storage medium. The method comprises the following steps: firstly, encoding text data in a potential safety hazard knowledge base to obtain a text feature vector; performing visual coding on the image by using the pre-trained feature mapping network to obtain local and global feature vectors; text knowledge recall is carried out through the vectors to obtain target text knowledge, after a target Prompt template is generated, an image and the template are input into a pre-training multi-modal large model, finally, a potential safety hazard recognition result is obtained, and the accuracy and efficiency of potential safety hazard recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, computer equipment and readable storage medium for performing knowledge retrieval based on images to enhance the identification of safety hazards. Background Art

[0002] In the field of safety hazard identification, traditional methods often rely on single-modal data for analysis, which has problems such as low recognition accuracy and poor adaptability to complex scenarios. With the continuous enrichment of image data and related knowledge bases, how to effectively combine the two has become a research hotspot. However, there are modal differences between visual information and textual knowledge, which makes it difficult to directly associate and utilize them. Summary of the invention

[0003] The purpose of the present invention is to provide a method, device, computer equipment and readable storage medium for enhancing the identification of safety hazards through knowledge retrieval based on pictures.

[0004] In a first aspect, an embodiment of the present invention provides a method for enhancing the identification of safety hazards by performing knowledge retrieval based on images, comprising:

[0005] Text encoding is performed on text data in a safety hazard knowledge base corresponding to the image to be identified to obtain multiple text feature vectors;

[0006] Visually encode the image to be recognized based on a pre-trained feature mapping network to obtain a local feature vector and a global feature vector corresponding to the image to be recognized; the text feature vector, the local feature vector and the global feature vector are in a feature vector space of the same dimension;

[0007] Performing text knowledge recall on the multiple text feature vectors according to the local feature vectors and the global feature vector to obtain target text knowledge;

[0008] Generate a target Prompt template based on the target text knowledge;

[0009] The image to be identified and the target prompt template are input into a pre-trained multimodal large model to obtain a safety hazard identification result of the image to be identified.

[0010] In a possible implementation, the feature mapping network is obtained through the following training, including:

[0011] Get sample text;

[0012] Replacing keywords in the sample text with preset placeholders to obtain mask text, wherein the keywords are keywords related to potential safety hazards;

[0013] Inputting the sample text and the mask text into an initial text encoder for encoding respectively to obtain a corresponding sample text feature vector and a mask text feature vector;

[0014] Based on the sample text feature vector, the mask text feature vector and Gaussian noise for bridging the modal gap between the visual feature space and the text feature space, through the objective function: Training is performed to obtain the trained feature mapping network;

[0015] in, is text encoding, η is Gaussian noise, T is sample text, T' is mask text, and l is MSE loss function.

[0016] In a possible implementation, performing local visual encoding on the image to be identified based on a pre-trained feature mapping network to obtain a local feature vector corresponding to the image to be identified includes:

[0017] Performing average segmentation processing on the image to be identified to obtain multiple local images;

[0018] The pre-trained feature mapping network is used to perform visual encoding on each of the local images to obtain a local feature vector corresponding to the image to be identified.

[0019] In a possible implementation, visually encoding the image to be identified based on a pre-trained feature mapping network to obtain a global feature vector corresponding to the image to be identified includes:

[0020] The pre-trained feature mapping network is used to perform global visual encoding on the image to be identified to obtain a local feature vector corresponding to the image to be identified.

[0021] In a possible implementation, the performing text knowledge recall on the multiple text feature vectors according to the local feature vectors and the global feature vector to obtain target text knowledge includes:

[0022] Respectively calculating the similarity between each of the text feature vectors and the local feature vector and the global feature vector to obtain a matching score between each of the text feature vectors and the local feature vector and the global feature vector;

[0023] Determining a target text feature vector from the plurality of text feature vectors according to a preset score threshold;

[0024] Based on the target text feature vector, text knowledge is recalled from the potential safety hazard knowledge base to obtain target text knowledge.

[0025] In a possible implementation, determining a target text feature vector from the multiple text feature vectors according to a preset score threshold includes:

[0026] Normalizing the matching score to obtain a normalized score;

[0027] Comparing the normalized score with a preset score threshold, and selecting text feature vectors with scores higher than the score threshold as candidate text feature vectors;

[0028] Cluster analysis is performed according to the semantic relevance of the candidate text feature vectors, and a representative text feature vector is selected from each cluster as the target text feature vector.

[0029] In a possible implementation, respectively calculating the similarity between each of the text feature vectors and the local feature vector and the global feature vector includes:

[0030] Using ColBERT through the formula: Calculating the similarity between each of the text feature vectors and the local feature vector and the global feature vector respectively;

[0031] Where, Q = [F M ([φ(r 1 ),φ(r 2 ),...,φ(r 9 )]),F M (φ(I))],D=F L (d) is the text feature vector, F M ([φ(r 1 ),φ(r 2 ),....,φ(r 9 )]) is the local eigenvector, F M (φ(I)) is the global eigenvector.

[0032] In a second aspect, an embodiment of the present invention provides a device for enhancing the identification of safety hazards by performing knowledge retrieval based on images, including:

[0033] The encoding module is used to perform text encoding on the text data in the safety hazard knowledge base corresponding to the image to be identified to obtain a plurality of text feature vectors; perform visual encoding on the image to be identified based on a pre-trained feature mapping network to obtain a local feature vector and a global feature vector corresponding to the image to be identified; the text feature vector, the local feature vector and the global feature vector are in a feature vector space of the same dimension;

[0034] The recognition module is used to recall text knowledge of the multiple text feature vectors according to the local feature vectors and the global feature vectors to obtain target text knowledge; generate a target Prompt template based on the target text knowledge; input the image to be recognized and the target Prompt template into a pre-trained multimodal large model to obtain a safety hazard recognition result of the image to be recognized.

[0035] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device executes the method described in the first aspect.

[0036] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method described in the first aspect.

[0037] Compared with the prior art, the beneficial effects provided by the present invention include: using the method, device, computer equipment and readable storage medium disclosed by the present invention to enhance the identification of safety hazards through knowledge retrieval based on pictures, obtain text feature vectors by encoding text data in the safety hazard knowledge base, and then use the pre-trained feature mapping network to visually encode the image to obtain local and global feature vectors. Through these vectors, text knowledge is recalled to obtain target text knowledge, and after generating the target prompt template, the image and template are input into the pre-trained multimodal large model, and finally the safety hazard identification result is obtained, which improves the accuracy and efficiency of safety hazard identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0039] Figure 1 A schematic diagram of the steps of a method for enhancing the identification of safety hazards through image-based knowledge retrieval provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of an intersection framework for enhancing the identification of safety hazards through image-based knowledge retrieval provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a training framework for a feature mapping network provided in an embodiment of the present invention;

[0042] Figure 4 A schematic block diagram of the structure of a device for performing knowledge retrieval based on images to enhance the identification of potential safety hazards provided by an embodiment of the present invention;

[0043] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0045] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0046] In order to solve the technical problems in the aforementioned background technology, Figure 1 A flow chart of a method for enhancing the identification of safety hazards through knowledge retrieval based on images provided in an embodiment of the present disclosure is provided below. The method for enhancing the identification of safety hazards through knowledge retrieval based on images is introduced in detail.

[0047] Step S201, performing text encoding on the text data in the safety hazard knowledge base corresponding to the image to be identified to obtain a plurality of text feature vectors;

[0048] Step S202, visually encoding the image to be recognized based on a pre-trained feature mapping network to obtain a local feature vector and a global feature vector corresponding to the image to be recognized; the text feature vector, the local feature vector and the global feature vector are in a feature vector space of the same dimension;

[0049] Step S203, performing text knowledge recall on the multiple text feature vectors according to the local feature vectors and the global feature vector to obtain target text knowledge;

[0050] Step S204, generating a target Prompt template based on the target text knowledge;

[0051] Step S205: input the image to be identified and the target prompt template into a pre-trained multimodal large model to obtain a safety hazard identification result of the image to be identified.

[0052] In an embodiment of the present invention, for example, it is assumed that the server is processing a task on identifying safety hazards in industrial plants. The server stores a huge knowledge base of safety hazards, which contains various text descriptions related to plant safety, such as "exposed wires in the plant pose a risk of electric shock", "piling debris in the fire passage affects evacuation", "no guardrails are set up for high-altitude operations, which is prone to fall", etc. The server must first preprocess these text data. For example, the text may be cleaned to remove some unnecessary punctuation or special characters, and then operations such as stem extraction or word form restoration are performed to ensure the normalization of the text. Then, the server uses a pre-selected text encoder (such as the M3E model) to encode these preprocessed texts. Taking the text "exposed wires in the plant pose a risk of electric shock" as an example, the text encoder will convert it into a text feature vector of a specific dimension (assuming it is d-dimensional). This feature vector represents the semantic information of this text in a high-dimensional space. For each text in the knowledge base, the server will perform the same operation to obtain multiple text feature vectors. These text feature vectors are like business cards, each of which represents a piece of text information about safety hazards in the knowledge base, and they are all in the same high-dimensional feature vector space, which is convenient for subsequent comparison and calculation of correlation with image feature vectors. ##Second, visually encode the image to be identified based on a pre-trained feature mapping network to obtain the local feature vector and global feature vector corresponding to the image to be identified; the text feature vector, the local feature vector and the global feature vector are in the feature vector space of the same dimension.

[0053] The server receives an image of the interior of a factory to be identified. This image contains many elements, such as various equipment, channels, workers, etc. The server first performs an average segmentation process on this image. According to the settings, it is segmented into 3*3 image blocks, thus obtaining 9 partial images.

[0054] Take one of the local images as an example. This local image can be a corner in a factory with some wires and distribution boxes. The server uses a pre-trained feature mapping network to visually encode this local image. This feature mapping network is trained with a large amount of data before, and it can convert the visual information of this local image into a feature vector. This feature vector is in the same dimensional feature vector space as the previous text feature vector. The server will perform the same operation on the other 8 local images, and finally obtain 9 local feature vectors, each of which captures the detailed features of the corresponding local area in the image. For example, one of the local feature vectors may focus on reflecting the layout and status of the wires and other visual information related to safety hazards.

[0055] In addition to local encoding of the image, the server also needs to encode the global features of the entire image. For the image of the interior of the factory, the server uses the same pre-trained feature mapping network to perform global visual encoding on the entire image. In this process, the feature mapping network will comprehensively consider all elements in the image, such as the overall layout of the equipment, the direction of the channel, the distribution of personnel, etc., and then generate a global feature vector representing the overall semantic information of the entire image. This global feature vector is in the same dimensional feature vector space as the local feature vector and text feature vector obtained previously. For example, this global feature vector may reflect macroscopic visual information such as the overall order and safety atmosphere inside the entire factory.

[0056] The server now has 9 local feature vectors, 1 global feature vector, and multiple text feature vectors obtained previously. The server uses the ColBERT algorithm to calculate the similarity of each text feature vector with the local feature vector and the global feature vector.

[0057] For example, for the text feature vector corresponding to the text "Exposed wires in the factory pose a risk of electric shock", the server will compare it with 9 local feature vectors and 1 global feature vector respectively. Taking one of the local feature vectors (corresponding to the local image containing wires) as an example, the similarity between them is calculated using the ColBERT formula. If the calculated similarity is high, it means that the visual content of this local image is semantically related to the text "Exposed wires in the factory pose a risk of electric shock". For other text feature vectors and combinations of local feature vectors and global feature vectors, the server will repeat this calculation process to obtain the matching score of each text feature vector with the local feature vector and the global feature vector.

[0058] After the server obtains all the matching scores, it first normalizes these scores. For example, if the matching scores of a text feature vector with the local feature vector and the global feature vector are 0.6 and 0.7 respectively, the new scores are obtained after normalization.

[0059] Assuming that the preset score threshold is 0.5, the server compares the normalized score with this threshold. For text feature vectors with scores higher than 0.5, they are used as candidate text feature vectors. For example, after comparison, there are 10 text feature vectors with scores higher than the threshold, and these become candidate text feature vectors.

[0060] The server performs cluster analysis on these 10 candidate text feature vectors based on semantic relevance. For example, three of the text feature vectors are related to fire protection (such as "fire passages are piled with debris that affect evacuation", "fire extinguishers are expired and cannot be used normally", and "fire hydrants are blocked"), and they may be classified into one category; the other four text feature vectors related to electrical safety (such as "exposed wires in the factory pose a risk of electric shock", "distribution boxes are not grounded", "wires are aging", and "sockets are short-circuited") will be classified into another category, and so on.

[0061] Then, a representative text feature vector is selected from each cluster as the target text feature vector. For example, "Miscellaneous items piled up in fire passages affect evacuation" is selected as the representative text feature vector from the cluster related to fire protection, and "Exposure of wires in the factory poses a risk of electric shock" is selected as the representative text feature vector from the cluster related to electrical safety.

[0062] The server recalls text knowledge from the safety hazard knowledge base based on the determined target text feature vector. For example, for the target text feature vector "exposed wires in the factory pose a risk of electric shock", the server directly obtains the complete knowledge content of this text from the knowledge base, including possible solutions (such as timely wrapping of wires, replacement of damaged wire sheaths, etc.) and relevant knowledge such as hazard levels. The same operation is performed for other target text feature vectors, and the target text knowledge is finally obtained.

[0063] The server has obtained the target text knowledge, for example, the target text knowledge includes "exposed wires in the factory pose a risk of electric shock" and related solutions, etc. The server will embed these text contents into the predefined prompt template.

[0064] Assume that the predefined prompt template is "Please identify the safety hazards in the image based on the following information: [target text knowledge content], and the image is [image to be identified]". Then, for the target text knowledge "exposed wires in the factory pose a risk of electric shock", the generated target prompt template can be "Please identify the safety hazards in the image based on the following information: exposed wires in the factory pose a risk of electric shock, and the relevant solutions include timely wrapping of wires, replacing damaged wire sheaths, etc., and the image is [image inside the factory to be identified]". This target prompt template is designed to interact effectively with the multimodal large model.

[0065] The server inputs the image to be recognized with the interior scene of the factory and the target prompt template just generated into a pre-trained multimodal large model (such as a multimodal model such as CLIP or BLIP).

[0066] After receiving the image and the Prompt template, the multimodal large model will comprehensively consider the visual information in the image and the textual knowledge information in the Prompt template. Taking exposed wires as an example, the model will look for visual features related to "exposed wires" in the image, while combining relevant knowledge in the Prompt template (such as electric shock risks, solutions, etc.). If exposed wires are indeed found in the image, the model will output a recognition result containing "there are safety hazards of exposed wires in the factory", and may also give some relevant content such as hazard levels and solution suggestions based on the information in the Prompt template. If no safety hazards related to the target textual knowledge are found in the image, the model will output a result that no relevant safety hazards are found.

[0067] In the embodiment of the present invention, the feature mapping network is obtained through the following training, and can be implemented through the following examples.

[0068] Get sample text;

[0069] Replacing keywords in the sample text with preset placeholders to obtain mask text, wherein the keywords are keywords related to potential safety hazards;

[0070] Inputting the sample text and the mask text into an initial text encoder for encoding respectively to obtain a corresponding sample text feature vector and a mask text feature vector;

[0071] Based on the sample text feature vector, the mask text feature vector and Gaussian noise for bridging the modal gap between the visual feature space and the text feature space, through the objective function: Training is performed to obtain the trained feature mapping network;

[0072] in, is text encoding, η is Gaussian noise, T is sample text, T' is mask text, and l is MSE loss function.

[0073] In the embodiment of the present invention, the server is responsible for training a feature mapping network, which will be used for image-based security risk identification. First, the server obtains sample texts from a large security-related text database. These sample texts are all related to security risks and cover security issues that may occur in various scenarios.

[0074] For example, sample texts include "Failure to set up warning signs at construction sites can easily lead to accidents", "Poor ventilation in chemical workshops may cause toxic gas accumulation", "Goods stacked too high in warehouses are at risk of collapse", etc. These sample texts are collected from a large number of safety specification documents, accident reports, safety inspection records and other materials. They represent different types of safety hazards and provide a rich source of semantic information for the training of feature mapping networks.

[0075] The server performs a keyword replacement operation on each sample text obtained. Take the sample text "Failure to set up warning signs at construction sites can easily lead to accidents" as an example, where "construction sites" and "warning signs" are keywords closely related to safety hazards.

[0076] According to the preset rules, the server replaces these keywords with the preset placeholder "[$]", so this sample text is converted to "[$] Not setting [$] is likely to cause accidents", which is the mask text. For other sample texts, such as "Poor ventilation in chemical workshops may cause toxic gas accumulation", the keywords "chemical workshop" and "poor ventilation" are replaced to obtain mask text such as "[$][$] may cause toxic gas accumulation". In this way, the server processes all sample texts and obtains the corresponding mask text. The purpose of this is to enable the model to better understand the structure and semantic relationship of the text in subsequent training. It also helps to reduce the excessive impact of specific keywords on model training, so that the model can handle different safety hazard scenarios more generally.

[0077] The server uses a pre-selected initial text encoder (such as the M3E model) to encode the sample text and the mask text. For the sample text "The lack of warning signs at construction sites can easily lead to accidents", the initial text encoder converts it into a sample text feature vector of a specific dimension (assuming it is d-dimensional). This feature vector represents the semantic information of this sample text in a high-dimensional space, including information about the relationship between concepts such as construction sites, warning signs, and accidents.

[0078] At the same time, for the corresponding mask text "[$]Not setting [$] is likely to lead to accidents", the initial text encoder will also convert it into a mask text feature vector of the same dimension d. Although this mask text feature vector has lost some specific information due to the replacement of keywords, it still retains the structure of the text and some semantic information. For example, it can still reflect the semantic structure that failure to set something at a certain location may lead to accidents. The server will perform the same encoding operation on all sample texts and mask texts to obtain the corresponding sample text feature vectors and mask text feature vector sets.

[0079] When training, the server needs to consider bridging the modal gap between the visual feature space and the text feature space. To this end, the server adds Gaussian noise to the sample text feature vector. By consulting relevant literature and conducting a large number of experiments, the server determines the standard deviation of the Gaussian distribution (for example, setting it to a suitable value).

[0080] Taking the previously obtained sample text feature vector corresponding to "Failure to set up warning signs at construction sites can easily lead to accidents" as an example, the server adds Gaussian noise to this feature vector according to the selected Gaussian distribution. The feature vector after adding noise will change to a certain extent, but this change is within a controllable range. The purpose is to make this feature vector more robust and help to narrow the distance with the visual feature space. The server will perform the same operation for other sample text feature vectors.

[0081] The server calculates the loss based on the objective function (MSE loss function). For each pair of sample text feature vectors (after adding Gaussian noise) and mask text feature vectors, the server substitutes them into the objective function for calculation. For example, for a pair of feature vectors, the MSE loss value between them is calculated.

[0082] The server then uses an optimization algorithm (such as stochastic gradient descent) to minimize this loss function. During the training process, the parameters of the model are constantly adjusted. As the training progresses, the gap between the sample text feature vector (after adding Gaussian noise) and the masked text feature vector gradually decreases, which means that the feature mapping network is able to better process text features and to some extent bridge the modal gap between the visual feature space and the text feature space.

[0083] After multiple rounds of training, when the loss function converges to a satisfactory level, the server obtains a trained feature mapping network. This trained feature mapping network can be used for subsequent visual encoding operations of the image to be identified, thus playing an important role in the task of enhancing the identification of safety hazards through knowledge retrieval based on images.

[0084] In an embodiment of the present invention, local visual encoding is performed on the image to be identified based on a pre-trained feature mapping network to obtain a local feature vector corresponding to the image to be identified, which can be implemented through the following examples.

[0085] Performing average segmentation processing on the image to be identified to obtain multiple local images;

[0086] The pre-trained feature mapping network is used to perform visual encoding on each of the local images to obtain a local feature vector corresponding to the image to be identified.

[0087] In an embodiment of the present invention, exemplarily, the server receives an image to be identified, which is an internal scene of a large factory workshop, containing many complex elements, such as various large-scale production equipment, worker operation areas, material storage areas, passages, and electrical lines, etc. These elements are distributed in different locations throughout the workshop.

[0088] In order to obtain the local features of the image, the server performs an average segmentation process on the image to be identified according to a predetermined rule. Assuming that the server is set to segment the image into a 3×3 grid, the image will be evenly segmented into 9 local images.

[0089] Take one of the partial images as an example. This partial image may just cover an electrical control cabinet and a small area around it. In this partial image, you can see part of the cabinet door, some indicator lights, some wires connecting the control cabinet, and a small part of the surrounding ground. Other partial images also contain different local scenes in the workshop. For example, some partial images are scenes of workers operating in front of a certain equipment, and some are scenes of materials stacked in the corner. Through this average segmentation process, the server decomposes the complex overall image into multiple relatively simple sub-images with clear local semantics, providing a basis for subsequent local visual encoding.

[0090] The server uses a pre-trained feature mapping network to visually encode each local image obtained by segmentation. Taking the local image containing the electrical control cabinet mentioned earlier as an example, the feature mapping network begins to process this local image.

[0091] This feature mapping network is pre-trained with a large amount of different image data, and it can recognize various visual features in the image. When processing this local image, the network will capture the shape, color, texture and other features of the electrical control cabinet, the on and off status of the indicator light, the direction of the wires and other detailed information. After a series of complex calculations and transformations, the feature mapping network converts the visual information of this local image into a local feature vector of a specific dimension (assuming it is d-dimensional). This local feature vector is like a unique identifier of this local image in a high-dimensional feature space. It encodes all the visual information related to safety hazards in this local image. For example, the appearance characteristics of electrical equipment may indicate whether the equipment is operating normally, whether the direction of the wires meets safety regulations, etc.

[0092] The server performs visual encoding on the remaining eight partial images in the same way. For example, for partial images containing scenes of workers operating equipment, the feature mapping network will focus on elements such as the worker's posture, the equipment's operating interface status, and surrounding safety protection facilities, and convert these visual information into corresponding local feature vectors. For partial images of material stacking corners, the network will capture information such as the material stacking height, stacking method, and whether it is close to the channel, and generate corresponding local feature vectors.

[0093] After encoding all nine local images, the server obtains nine local feature vectors, which together constitute the set of local feature vectors corresponding to the image to be identified. These local feature vectors, together with the global feature vectors, will be used to recall relevant textual knowledge from the safety hazard knowledge base, thereby providing a rich visual information basis for identifying safety hazards in images.

[0094] In an embodiment of the present invention, visual encoding is performed on the image to be identified based on a pre-trained feature mapping network to obtain a global feature vector corresponding to the image to be identified, which can be implemented through the following examples.

[0095] The pre-trained feature mapping network is used to perform global visual encoding on the image to be identified to obtain a local feature vector corresponding to the image to be identified.

[0096] In an embodiment of the present invention, illustratively, the server receives an image of the interior of a large shopping mall to be identified, and the image covers a vast space, including multiple floors, numerous stores, customers, elevators, stairs, fire-fighting facilities, and a complex channel layout.

[0097] The server uses a pre-trained feature mapping network to start global visual encoding of the entire image to be recognized. This pre-trained feature mapping network is trained on a large number of different scenes and different types of image data, and it has strong visual feature extraction capabilities.

[0098] When performing global visual encoding on the interior image of a large shopping mall, the feature mapping network considers all elements in the image at the same time. For example, it pays attention to the overall architectural structure of the mall, including the connection between floors, the size and shape of different areas, etc. For stores, it captures information such as the distribution of stores, store types (catering, clothing, or other types), and the flow of people at the store entrance.

[0099] At the same time, the network will not ignore key elements such as elevators and stairs. It will analyze the operating status of the elevator (whether it is rising or falling, whether the elevator door is closed normally, etc.), the width of the stairs and the flow of people, and other safety-related visual features. In terms of fire protection facilities, it will check whether the location of fire extinguishers and fire hydrants is obvious and whether they are blocked. As for the distribution of customers, it can analyze densely populated areas and sparsely populated areas, which is very important for judging safety hazards (such as the risk of crowding and trampling).

[0100] In addition, the network focuses on channel layout, which determines whether the channel is unobstructed, whether there are obstacles blocking it, and whether the width of the channel meets safety standards.

[0101] After a comprehensive and detailed analysis of the entire image, the feature mapping network integrates all of this visual information and, through a series of complex calculations and transformations, ultimately generates a global feature vector of a specific dimension (assuming it is d-dimensional). This global feature vector is like a comprehensive identifier of the interior image of the shopping mall in a high-dimensional feature space, encoding all global visual information related to safety hazards in the entire image.

[0102] For example, this global feature vector may reflect the overall sense of order in the mall. If some values ​​in the vector indicate that the stores are chaotically distributed, the passages are narrow and crowded, then this may indicate a higher safety risk, such as difficulty in evacuating in case of fire, prone to stampede accidents, etc. On the other hand, if the values ​​in the vector show that the fire-fighting facilities are reasonably and obviously located, the passages are unobstructed, and the personnel are evenly distributed, then the overall safety situation may be relatively good.

[0103] This global feature vector, together with the previously obtained local feature vector, will be used to recall relevant textual knowledge from the safety hazard knowledge base, thereby providing a comprehensive visual information basis for accurately identifying safety hazards in this image of the interior of the shopping mall. In this way, both local detailed safety hazards (such as electrical problems in a store, which can be reflected by local feature vectors) and macro safety hazards involving the entire shopping mall (such as overall evacuation capacity, which is reflected by global feature vectors) can be effectively identified.

[0104] In the embodiment of the present invention, the text knowledge recall of the multiple text feature vectors according to the local feature vectors and the global feature vector to obtain the target text knowledge can be implemented through the following examples.

[0105] Respectively calculating the similarity between each of the text feature vectors and the local feature vector and the global feature vector to obtain a matching score between each of the text feature vectors and the local feature vector and the global feature vector;

[0106] Determining a target text feature vector from the plurality of text feature vectors according to a preset score threshold;

[0107] Based on the target text feature vector, text knowledge is recalled from the potential safety hazard knowledge base to obtain target text knowledge.

[0108] In the embodiment of the present invention, the server has obtained the local feature vector and the global feature vector corresponding to the image to be identified (for example, the image inside a school building), and also has multiple text feature vectors in the safety hazard knowledge base. Now, the similarity is calculated to obtain the matching score.

[0109] Take a text feature vector in the knowledge base as an example. The text corresponding to this text feature vector is "The lighting facilities in the stairwell of the teaching building are damaged, posing a risk of falling." The server uses the ColBERT algorithm to calculate the similarity between this text feature vector and the local feature vector of the image.

[0110] Assume that the image is segmented into 9 partial images, each of which has a corresponding local feature vector. For one of the local feature vectors, such as the local feature vector corresponding to the stairwell partial image, the server calculates the ColBERT algorithm as follows:

[0111] First, the algorithm extracts semantic information related to the stairwell from the text feature vector, such as the representation of concepts such as "stairwell", "lighting facilities", and "risk of falling" in the high-dimensional vector space. At the same time, the algorithm extracts the visual feature representation of the stairwell from the local feature vector of the stairwell, such as the representation of visual information such as the shape, color, and surrounding environment of the stairs in the feature vector. Then, according to the similarity calculation formula of ColBERT, these semantic information and visual information are compared and calculated to obtain a preliminary similarity score, such as 0.6.

[0112] This calculation process is repeated for the other 8 local feature vectors to obtain the similarity scores of 9 local feature vectors related to this text feature vector.

[0113] Next, the server calculates the similarity between the text feature vector and the global feature vector of the image. For images of the interior of a school building, the global feature vector contains information such as the overall layout of the building, the distribution of various areas, and the overall flow of personnel.

[0114] The server also uses the ColBERT algorithm to extract semantic information about the overall safety hazards of the teaching building from the text feature vector and extract the overall visual feature representation of the teaching building from the global feature vector. Then, the calculation is performed, assuming that the obtained similarity score is 0.5.

[0115] In order to obtain the matching score of the text feature vector with the local feature vector and the global feature vector, the server will adopt a comprehensive strategy. For example, the larger value of the similarity score with the local feature vector (here 0.6) and the similarity score with the global feature vector (0.5), that is, 0.6, can be taken as the matching score of the text feature vector.

[0116] The server will repeat the above calculation process for each text feature vector in the knowledge base to obtain the matching score of each text feature vector with the local feature vector and the global feature vector.

[0117] After the server obtains the matching scores of all text feature vectors, it first performs normalization. For example, the matching score may range from 0 to 1. The original matching score of a text feature vector is 0.6, which may become 0.8 after normalization (assuming that the normalization calculation method is to divide the original score by the maximum value of all scores).

[0118] Assume that the preset score threshold is 0.7. The server compares the normalized score with the score threshold. For text feature vectors with scores higher than 0.7, they are used as candidate text feature vectors. For example, after comparison, there are 5 text feature vectors with scores higher than the threshold, and these become candidate text feature vectors.

[0119] The server performs cluster analysis on these five candidate text feature vectors based on semantic relevance. For example, two of the text feature vectors are related to firefighting facilities (such as "the fire hydrants in the teaching building are out of water and cannot extinguish fires", "fire extinguishers are expired and cannot be used normally"), and they may be classified into one category; the other three text feature vectors related to stair safety (such as "the lighting facilities in the stairwell of the teaching building are damaged and there is a risk of falling", "the handrails of the stairs are loose and there is a risk of falling", "the steps of the stairs are broken and easy to trip") will be classified into another category.

[0120] Then, a representative text feature vector is selected from each cluster as the target text feature vector. For example, "The fire hydrants in the teaching building have no water and cannot extinguish the fire" is selected as the representative text feature vector from the cluster related to firefighting facilities, and "The lighting facilities in the stairwell of the teaching building are damaged and there is a risk of falling" is selected as the representative text feature vector from the cluster related to stair safety.

[0121] The server recalls text knowledge from the safety hazard knowledge base based on the determined target text feature vector. For the target text feature vector "the fire hydrant in the teaching building has no water and cannot extinguish the fire", the server searches for the corresponding complete text knowledge in the safety hazard knowledge base. This complete text knowledge may include detailed reasons for the lack of water in the fire hydrant (such as long-term lack of inspection and maintenance, pipe rupture, etc.), possible serious consequences (such as the fire cannot be extinguished in time, causing the fire to spread, endangering the lives of teachers and students and the safety of the teaching building), and corresponding solutions (such as immediate inspection and maintenance of the pipe, refilling with water, etc.).

[0122] For another target text feature vector "The damaged lighting facilities in the stairwell of the teaching building pose a risk of falling", the server also obtains its complete text knowledge from the knowledge base, including common causes of damaged lighting facilities (such as damaged bulbs, circuit failures, etc.), specific assessments of the risk of falling (such as assessing the risk level based on factors such as the flow of people in the stairwell and the slope of the stairs), and solutions (such as timely replacement of bulbs, circuit repairs, etc.).

[0123] Through such text knowledge recall, the server obtains the target text knowledge, which will be used to generate the target Prompt template, thereby providing important text information basis for accurately identifying safety hazards in the interior images of the teaching building.

[0124] In the embodiment of the present invention, the step of determining the target text feature vector from the multiple text feature vectors according to the preset score threshold may be implemented through the following examples.

[0125] Normalizing the matching score to obtain a normalized score;

[0126] Comparing the normalized score with a preset score threshold, and selecting text feature vectors with scores higher than the score threshold as candidate text feature vectors;

[0127] Cluster analysis is performed according to the semantic relevance of the candidate text feature vectors, and a representative text feature vector is selected from each cluster as the target text feature vector.

[0128] In the embodiment of the present invention, the server has calculated the matching scores of each text feature vector with the local feature vector and the global feature vector. Assume that the server is processing a task about identifying potential safety hazards in a hospital, and obtains a large number of text feature vectors from the potential safety hazard knowledge base. The corresponding text content includes, for example, "the failure of the call system in the ward affects the patient's call for help", "the substandard disinfection in the operating room is prone to infection", "the clutter in the corridor hinders the passage of stretchers", etc., and the matching scores of these text feature vectors are calculated for an image to be identified inside the hospital.

[0129] Taking the text feature vector corresponding to "The call system failure in the ward affects the patient's call for help" as an example, its matching score is 0.6, and the maximum matching score of all text feature vectors is 0.8. The server uses a simple normalization method, that is, dividing the matching score of each text feature vector by the maximum value, so the normalized score of this text feature vector is 0.6 / 0.8=0.75.

[0130] For the text feature vector corresponding to "Unsatisfactory disinfection of operating rooms can easily lead to infection", assuming that its matching score is 0.5, the normalized score is 0.5 / 0.8=0.625. The server will normalize all calculated matching scores in this way to obtain the normalized score of each text feature vector. The purpose of this is to unify all scores to the same scale, which is convenient for subsequent comparison with the preset score threshold.

[0131] The server pre-sets a score threshold, assuming that this threshold is 0.7. After normalization, the server begins to compare the normalized score of each text feature vector with this threshold.

[0132] For the text feature vector corresponding to “the failure of the call system in the ward affects the patient's request for help” which has a score of 0.75 after normalization, since 0.75>0.7, this text feature vector is screened out as a candidate text feature vector.

[0133] As for the text feature vector corresponding to “Unsatisfactory disinfection of operating rooms can easily lead to infection” with a normalized score of 0.625, this text feature vector will not be selected because 0.625<0.7.

[0134] The server will traverse all normalized text feature vectors and select those with scores higher than 0.7 as candidate text feature vectors. For example, there is a text feature vector corresponding to the text "Unclear parking space markings in the hospital parking lot lead to parking chaos", and its normalized score is 0.72. This text feature vector will also be selected as a candidate text feature vector. Through this comparison and screening, the server determines a number of candidate text feature vectors that have a high correlation with the visual features of the image to be identified from a large number of text feature vectors. These candidate text feature vectors will serve as the basis for further analysis.

[0135] The server now performs a cluster analysis of semantic relevance on the selected candidate text feature vectors. Assume that five candidate text feature vectors have been selected through the previous steps, and the corresponding texts are "The call system failure in the ward affects the patient's call for help", "The oxygen supply equipment failure in the ward endangers the patient's life", "The unclear parking space markings in the hospital parking lot lead to parking chaos", "The emergency call button in the hospital elevator car is out of order", and "The chaotic placement of medicines in the hospital pharmacy affects the efficiency of taking medicines".

[0136] By analyzing the semantic content of these texts, the server found that "the failure of the call system in the ward affects the patient's call for help" and "the failure of the oxygen supply equipment in the ward endangers the patient's life" are both related to the failure of medical equipment in the ward. These two text feature vectors have a high semantic correlation, so they are clustered into one category.

[0137] However, "The unclear parking space markings in the hospital parking lot lead to parking chaos" has a large semantic difference with the other texts and forms a separate category. "The emergency call button in the hospital elevator car is out of order" is different from the equipment failure in the ward, but both belong to the category of hospital facility failure. Therefore, the two text feature vectors related to the ward are related to each other under a large semantic category and are also included in the cluster of ward equipment failure. "The chaotic placement of medicines in the hospital pharmacy affects the efficiency of taking medicines" forms a separate category because it mainly involves the management of the pharmacy.

[0138] Select a representative text feature vector from each cluster as the target text feature vector. For the cluster of ward equipment failure, the server selects "the ward call system failure affects the patient's call for help" as the representative text feature vector. This is because this text in this cluster typically reflects the impact of ward equipment failure on patient safety.

[0139] For the parking lot related cluster, the only text feature vector "Unclear markings of parking spaces in the hospital parking lot lead to parking chaos" is directly used as the representative text feature vector of the cluster.

[0140] For the clustering of pharmacy management, "the disordered placement of medicines in hospital pharmacies affects the efficiency of taking medicines" is used as a representative text feature vector. In this way, the server determines the target text feature vectors from the candidate text feature vectors. These target text feature vectors can more accurately represent the safety hazard knowledge related to the image to be identified, and provide key text information basis for subsequent safety hazard identification work.

[0141] In the embodiment of the present invention, respectively calculating the similarity between each of the text feature vectors and the local feature vector and the global feature vector may be implemented through the following examples.

[0142] Using ColBERT through the formula: Calculating the similarity between each of the text feature vectors and the local feature vector and the global feature vector respectively;

[0143] Where, Q = [F M ([φ(r 1 ),φ(r 2 ),...,φ(r 9 )]),F M (φ(I))],D=F L (d) is the text feature vector, F M ([φ(r 1 ),φ(r 2 ),....,φ(r 9 )]) is the local eigenvector, F M (φ(I)) is the global eigenvector.

[0144] In the embodiment of the present invention, for example, it is assumed that the server is processing a task on identifying potential safety hazards in an airport terminal building. The server has obtained multiple text feature vectors in the potential safety hazard knowledge base, and has processed the airport terminal building image to be identified to obtain a local feature vector.

[0145] Take a text feature vector as an example. The text corresponding to this text feature vector is "There is a risk of electric shock due to exposed wires near seats in the waiting hall." This text feature vector represents the semantic information about exposed wires near seats in the waiting hall in a high-dimensional space.

[0146] At the same time, it is assumed that the image is segmented into multiple local images, wherein the local feature vector of one local image covers the visual information of a certain seat area in the terminal hall, including the seat layout, surrounding facilities, ground conditions, etc.

[0147] When calculating similarity, ColBERT first preprocesses the text feature vector and the local feature vector. For the text feature vector, it analyzes and extracts the representation of each semantic element in "Exposure wires near seats in the waiting hall pose a risk of electric shock" (such as "waiting hall", "seats", "exposure wires", "risk of electric shock") in high-dimensional space. For the local feature vector, it extracts the representation of visual elements related to the seat area (such as the shape and color of the seat, the outline of nearby objects, etc.) in the feature vector.

[0148] Then, according to the formula, the server uses ColBERT to substitute these representations into the formula for calculation. For example, for the semantic element "exposed wires" in the text feature vector, ColBERT will look for the best matching visual element representation in the local feature vector, which can be some lines similar to the shape of wires or some visual features of electrical equipment. By matching and calculating these elements, a similarity score is finally obtained. Assuming that this similarity score is 0.6, it means that this text feature vector has a certain degree of correlation with this local feature vector.

[0149] The server will repeat the above calculation process for this text feature vector and all local feature vectors. For example, for another local feature vector, it corresponds to a local image near the boarding gate of the terminal hall. This local feature vector contains visual information such as boarding gate facilities, signs, and passenger queues.

[0150] ColBERT again uses the above calculation logic to analyze and calculate the text feature vector and this local feature vector. Since this local feature vector has little semantic relevance to the text "exposed wires near seats in the waiting hall", the calculated similarity score may be low, assuming it is 0.1.

[0151] The server will calculate each text feature vector in the potential safety hazard knowledge base and each local feature vector of the image one by one in the above manner, thereby obtaining a similarity score between each text feature vector and all local feature vectors.

[0152] Let's continue with the example of the text feature vector "There is a risk of electric shock due to exposed wires near the seats in the terminal hall". Now we need to calculate its similarity with the global feature vector of the airport terminal image.

[0153] The global feature vector of an airport terminal building includes comprehensive visual information such as the overall architectural structure of the terminal building (such as the shape of the hall, number of floors, channel layout, etc.), the overall personnel distribution (such as the gathering area of ​​passengers, the distribution of staff, etc.), and the overall layout of major facilities (such as the distribution of boarding gates, security check areas, shops, etc.).

[0154] ColBERT also preprocesses the text feature vector and the global feature vector. For the text feature vector, it analyzes and extracts the representation of semantic elements such as "waiting hall", "seats", "exposed wires", and "risk of electric shock" in high-dimensional space. For the global feature vector, it extracts the representation of visual elements related to the overall layout and safety of the terminal building (such as the overall layout of the hall's electrical facilities, the relationship between crowded areas and electrical equipment, etc.) in the feature vector.

[0155] Then the calculation is performed according to the formula. During this calculation process, since the global feature vector covers the information of the entire terminal, ColBERT needs to find the match between text semantic elements and visual elements at a more macro level. For example, although the text is about exposed wires near the seats, from a global perspective, it is necessary to consider factors such as the relationship between the electrical system layout of the entire terminal and the seating area, as well as the impact of the flow of people in the terminal on the risk of electric shock. After complex calculations, it is assumed that the similarity score obtained is 0.5, indicating that this text feature vector has a certain degree of correlation with the global feature vector.

[0156] The server will repeat the above process of calculating similarity with the global feature vector for each text feature vector in the safety hazard knowledge base. For example, for another text feature vector corresponding to the text "the fire escape of the terminal building is occupied, affecting evacuation", ColBERT will follow the same calculation logic, first pre-process this text feature vector and the global feature vector, and then calculate the similarity between them according to the formula. In this way, the server obtains the similarity score between each text feature vector and the global feature vector.

[0157] Through the above detailed calculation process, the server uses ColBERT to calculate the similarity between each text feature vector and the local feature vector and the global feature vector according to the given formula, providing an important data basis for subsequent text knowledge recall and safety hazard identification.

[0158] In order to more clearly describe the solution provided by the embodiment of the present invention, a more detailed implementation is provided below.

[0159] 1. Knowledge recall based on visual feature vectors

[0160] This paper proposes a text knowledge recall method based on visual feature vectors, which is particularly suitable for extracting relevant text knowledge from image data to assist in the detection of image safety hazards using multimodal models. Figure 2 The specific implementation steps are as follows:

[0161] Text feature vector generation: First, the text data in the knowledge base is preprocessed and encoded. By applying the text encoder, each piece of text data is converted into a corresponding feature vector D = F L (d), where d represents the content of the text. This step facilitates subsequent similarity calculation by converting the original text into points in a high-dimensional vector space.

[0162] Image segmentation and local feature vector generation: For the image to be recognized, it is first segmented and evenly divided into 3*3 image blocks to obtain 9 independent image regions. Then, the 9 image blocks are visually encoded respectively, and the feature vector F corresponding to each image block is generated through the mapping network. M ([φ(r 1 ),φ(r 2 ),....,φ(r 9 )]), where r i represents the i-th image block, and φ represents the encoding of the image. This process aims to capture the detailed features of the local area of ​​the image and provide a more refined feature description for the subsequent recall task.

[0163] Global feature vector generation: In addition to extracting local feature vectors, the global features of the entire image are encoded, and the global feature vector F of the original image is generated through the visual encoder and mapping network. M (φ(I)), where I represents the entire image and φ represents the encoding of the image. This step ensures the grasp of the overall semantic information of the image and provides a global perspective for comprehensive analysis.

[0164] Text knowledge recall: In order to recall text knowledge related to the image from the knowledge base, this paper proposes to use ColBERT to calculate the image feature vector Q = [F M ([φ(r 1 ),φ(r 2 ),...,φ(r 9 )]),F M (φ(I))] and the text feature vector D = F L (d) The similarity between them. The specific similarity calculation formula is:

[0165] Among them, by comparing the local feature vector and global feature vector of the image and the feature vector of the text, the maximum matching score between them is calculated to measure the relevance between the image and the text. The higher the similarity score, the stronger the relevance between the text and the image, and the more accurately the relevant text knowledge can be recalled from the knowledge base [T 1 ,T 2 ].

[0166] Generate prompt template: Once the textual knowledge related to the image is recalled, embed the textual content into a predefined prompt template. This template is designed to interact further with the multimodal large model. Combining the original image input to the model and the generated prompt template, the system can eventually identify potential safety hazards or other information in the image.

[0167] (ii) Only use text to train the mapping network

[0168] The present invention provides an improved visual-language model (VLM) training method, which significantly improves the performance of multimodal tasks by optimizing the collaboration between image encoders and text encoders. Specifically, the present invention uses a relatively mature visual encoder that has been trained in the prior art to encode images, and uses an equally mature text encoder to encode text, thereby effectively reducing the complexity of the training process and the required computing resources.

[0169] Traditional multimodal learning models, such as CLIP, rely on large-scale image data for contrastive learning. However, in practical applications, there is often a lack of image data, which makes it difficult to train models through contrastive learning. To solve this problem, the present invention proposes a mapping network strategy that is only trained on text, thereby bypassing the dependence on a large amount of image data. The core of this strategy is to replace the traditional image-text joint training by training a text feature mapping network, and at the same time use a method of fusing Gaussian noise to bridge the modal gap between the visual feature space and the text feature space.

[0170] In the specific implementation process of the present invention, a pre-trained visual-language model (such as CLIP or BLIP) is first used to map the input image and text into feature vector spaces of the same dimension. However, due to the inherent feature space difference between the visual modality and the text modality, that is, the visual feature space and the text feature space are not completely aligned, this modality gap will have an adverse effect on the final performance of the model, thereby limiting the application effect of the visual feature vector.

[0171] In order to effectively solve the above problems, the present invention proposes a solution to reduce the modal gap by injecting noise or adopting a projection-based alignment method. Specifically, the present invention adopts a noise addition strategy, that is, injecting Gaussian noise into the feature vector space, aiming to reduce the modal gap between visual features and text features. In addition, in order to further optimize the alignment process, the present invention studies the distribution form of Gaussian noise and finds a comparative Gaussian noise distribution function, thereby achieving significant improvements in the alignment of visual features and text features.

[0172] On this basis, the present invention not only overcomes the difficulty of model training caused by insufficient image data, but also significantly improves the overall performance of multimodal tasks by training only for text. This innovative training method enables efficient and accurate visual-language model training with less computing resources even when data resources are limited. In addition, the application of this method is also widely applicable and can achieve excellent performance in a variety of different multimodal tasks.

[0173] In summary, the present invention successfully overcomes the shortcomings of traditional methods by optimizing the collaboration between image and text encoders, using Gaussian noise to narrow the modality gap, and training only on text, and provides a new, efficient and reliable approach for training vision-language models.

[0174] (III) Mapping network training process

[0175] The present invention provides a specific implementation method for training a visual-language model (VLM), which optimizes the performance of the model by introducing text masks and Gaussian noise. Figure 3 The following is a detailed description of the specific implementation process:

[0176] First, in the text x c In the preprocessing stage, all "keywords" in the text to be processed are replaced with the predefined placeholder "[$]". For example, for the text "factory safety exit", its keywords include "factory" and "safety exit", so the text will be converted to "[$] of [$]". We call this converted text mask text x t .

[0177] During the encoding step, the original text and masked text will be input into the same text encoder to generate corresponding feature vectors. c The feature vector e generated by the text encoder c , set to mask text x t The feature vector e generated by the text encoder t The text encoder uses a well-trained model, such as M3E, which can effectively capture the semantic features of the text.

[0178] In order to further narrow the gap between the visual modality and the textual modality, this paper introduces a Gaussian noise-based strategy to achieve modality alignment. Specifically, Gaussian noise η is added to the text feature vector to make the feature vector more robust and help bridge the modality gap between the visual feature space and the text feature space. The standard deviation of the Gaussian distribution is set to ε. By consulting relevant literature and conducting a large number of experiments, this paper selects the standard deviation value The Gaussian noise η is used to ensure that the main features of the text are retained during the modal alignment process, while the negative impact of the modal gap on the model performance can be effectively weakened.

[0179] Next, the training objective of the present invention is to minimize the following loss function:

[0180] in represents text encoding, η is Gaussian noise, T is text, T' is masked text, and l is the MSE loss function;

[0181] By minimizing the above loss function, the present invention effectively trains the mapping network F M , so that the text features are more closely aligned with the visual features, thereby improving the overall performance of the multimodal task. In particular, since only the text part is trained during the training process, this method still shows good results when data resources are limited, and can effectively reduce the consumption of computing resources.

[0182] The training method based on text mask and Gaussian noise proposed in the present invention significantly improves the performance of the vision-language model and has high practical application value and broad application prospects.

[0183] The present invention has several significant advantages, the core of which is to accurately recall textual knowledge related to the image by combining local and global features of the image, thereby effectively supporting the execution of multimodal tasks. Compared with traditional methods, the present invention does not require additional text data. This feature greatly reduces the complexity of the model and makes the entire application process simpler and easier. It also greatly simplifies the human interaction process in practical applications. This non-interactive recognition method improves the automation level of the system.

[0184] Traditional alignment of different modalities usually requires a large amount of image and text data for training to ensure that the mapping between visual features and language features is accurate. However, acquiring and processing large-scale multimodal data is not only time-consuming and labor-intensive, but also extremely demanding on computing resources. In response to this pain point, the present invention proposes a method that only uses text for training, that is, by optimizing the mapping network of text features, effective alignment of visual features and text features is achieved. In this way, the training of the model no longer depends on huge image data, thereby overcoming the training difficulties caused by insufficient image data.

[0185] Please refer to Figure 4 , Figure 4 An apparatus 110 for enhancing the identification of potential safety hazards based on image knowledge retrieval is provided in an embodiment of the present invention, comprising:

[0186] The encoding module 1101 is used to perform text encoding on the text data in the safety hazard knowledge base corresponding to the image to be identified to obtain a plurality of text feature vectors; perform visual encoding on the image to be identified based on a pre-trained feature mapping network to obtain a local feature vector and a global feature vector corresponding to the image to be identified; the text feature vector, the local feature vector and the global feature vector are in a feature vector space of the same dimension;

[0187] The recognition module 1102 is used to perform text knowledge recall on the multiple text feature vectors according to the local feature vectors and the global feature vectors to obtain target text knowledge; generate a target Prompt template based on the target text knowledge; input the image to be recognized and the target Prompt template into a pre-trained multimodal large model to obtain a safety hazard recognition result of the image to be recognized.

[0188] It should be noted that the implementation principle of the aforementioned device 110 for enhancing the identification of safety hazards based on pictures can refer to the implementation principle of the aforementioned method for enhancing the identification of safety hazards based on pictures, which will not be repeated here. It should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be fully implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the device 110 for enhancing the identification of safety hazards based on pictures can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the device 110 for enhancing the identification of safety hazards based on pictures. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or an instruction in software form.

[0189] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0190] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned apparatus 110 for enhancing the identification of safety hazards through knowledge retrieval based on images. Figure 5 As shown, Figure 5 The computer device 100 provided in the embodiment of the present invention is a structural block diagram. The computer device 100 includes an apparatus 110 for enhancing the identification of potential safety hazards through knowledge retrieval based on pictures, a memory 111, a processor 112 and a communication unit 113.

[0191] In order to realize data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these components can be realized through one or more communication buses or signal lines. The device 110 for performing knowledge retrieval based on pictures to enhance the identification of safety hazards includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the device 110 for performing knowledge retrieval based on pictures to enhance the identification of safety hazards stored in the memory 111, such as the software function modules and computer programs included in the device 110 for performing knowledge retrieval based on pictures to enhance the identification of safety hazards.

[0192] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is running, it controls the computer device where the readable storage medium is located to execute the aforementioned device 110 for enhancing the identification of safety hazards through knowledge retrieval based on images.

[0193] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A method for enhancing the identification of safety hazards through knowledge retrieval based on images, characterized in that: include: Text encoding is performed on text data in a safety hazard knowledge base corresponding to the image to be identified to obtain multiple text feature vectors; Visually encode the image to be recognized based on a pre-trained feature mapping network to obtain a local feature vector and a global feature vector corresponding to the image to be recognized; the text feature vector, the local feature vector and the global feature vector are in a feature vector space of the same dimension; Performing text knowledge recall on the multiple text feature vectors according to the local feature vectors and the global feature vector to obtain target text knowledge; Generate a target Prompt template based on the target text knowledge; The image to be identified and the target prompt template are input into a pre-trained multimodal large model to obtain a safety hazard identification result of the image to be identified.

2. The method according to claim 1, characterized in that The feature mapping network is obtained through the following training, including: Get sample text; Replacing keywords in the sample text with preset placeholders to obtain mask text, wherein the keywords are keywords related to potential safety hazards; Inputting the sample text and the mask text into an initial text encoder for encoding respectively to obtain a corresponding sample text feature vector and a mask text feature vector; Based on the sample text feature vector, the mask text feature vector and Gaussian noise for bridging the modal gap between the visual feature space and the text feature space, through the objective function: Training is performed to obtain the trained feature mapping network; in, is text encoding, η is Gaussian noise, T is sample text, T' is mask text, and l is MSE loss function.

3. The method according to claim 1, characterized in that Performing local visual encoding on the image to be identified based on a pre-trained feature mapping network to obtain a local feature vector corresponding to the image to be identified, including: Performing average segmentation processing on the image to be identified to obtain multiple local images; The pre-trained feature mapping network is used to perform visual encoding on each of the local images to obtain a local feature vector corresponding to the image to be identified.

4. The method according to claim 1, characterized in that: Visually encoding the image to be identified based on a pre-trained feature mapping network to obtain a global feature vector corresponding to the image to be identified, including: The pre-trained feature mapping network is used to perform global visual encoding on the image to be identified to obtain a local feature vector corresponding to the image to be identified.

5. The method according to claim 1, characterized in that The step of performing text knowledge recall on the multiple text feature vectors according to the local feature vectors and the global feature vector to obtain target text knowledge includes: Respectively calculating the similarity between each of the text feature vectors and the local feature vector and the global feature vector to obtain a matching score between each of the text feature vectors and the local feature vector and the global feature vector; Determining a target text feature vector from the plurality of text feature vectors according to a preset score threshold; Based on the target text feature vector, text knowledge is recalled from the potential safety hazard knowledge base to obtain target text knowledge.

6. The method according to claim 5, characterized in that The step of determining a target text feature vector from the plurality of text feature vectors according to a preset score threshold comprises: Normalizing the matching score to obtain a normalized score; Comparing the normalized score with a preset score threshold, and selecting text feature vectors with scores higher than the score threshold as candidate text feature vectors; Cluster analysis is performed according to the semantic relevance of the candidate text feature vectors, and a representative text feature vector is selected from each cluster as the target text feature vector.

7. The method according to claim 5, characterized in that The respectively calculating the similarity between each of the text feature vectors and the local feature vector and the global feature vector comprises: Using ColBERT through the formula: Calculating the similarity between each of the text feature vectors and the local feature vector and the global feature vector respectively; Where, Q = [F M ([φ(r1),φ(r2),...,φ(r9)]),F M (φ(I))],D=F L (d) is the text feature vector, F M ([φ(r1),φ(r2),....,φ(r9)]) is the local eigenvector, F M (φ(I)) is the global eigenvector.

8. A device for enhancing the identification of safety hazards through knowledge retrieval based on images, characterized in that: include: The encoding module is used to perform text encoding on the text data in the safety hazard knowledge base corresponding to the image to be identified to obtain a plurality of text feature vectors; perform visual encoding on the image to be identified based on a pre-trained feature mapping network to obtain a local feature vector and a global feature vector corresponding to the image to be identified; the text feature vector, the local feature vector and the global feature vector are in a feature vector space of the same dimension; A recognition module, configured to perform text knowledge recall on the plurality of text feature vectors according to the local feature vectors and the global feature vectors to obtain target text knowledge; and generate a target Prompt template based on the target text knowledge; The image to be identified and the target prompt template are input into a pre-trained multimodal large model to obtain a safety hazard identification result of the image to be identified.

9. A computer device, characterized in that: The computer device comprises a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Images