Multi-mode identification method and device for dangerous liquid and computer program product

Abstract

The invention relates to a multi-mode identification method and device for dangerous liquid and a computer program product, and the method comprises the steps: generating text data for describing the attributes of the dangerous liquid according to the image data of CT imaging, and enabling the attributes to at least comprise liquid components and container materials; encoding the image data by using an image encoder to obtain a visual vector; using a text encoder to encode each attribute in the text data to obtain a text vector, and storing the text vector in a text vector library; and for the image data for learning, training a multi-modal model for identifying the dangerous liquid by using the visual vector and the text vector. According to the multi-modal identification method and device for the dangerous liquid, the influence of container materials, liquid components and the like in the CT scanning image can be considered, so that the identification accuracy of the dangerous liquid is improved.

Description

Technical Field

[0001] The present application relates to the field of CT image processing, and in particular, to a multimodal identification method, device and computer program product for hazardous liquids. Background Art

[0002] With the rapid development of society, public safety has received more and more attention, and higher requirements have been put forward for the accuracy and efficiency of safety detection of dangerous liquids in luggage. In recent years, by adopting intelligent security inspection recognition algorithms to classify and locate CT images, it is possible to achieve relatively accurate and rapid recognition of dangerous liquids under current circumstances. However, most of the existing image detection methods are based on learning projection images of CT data at different angles, mainly based on the shape and color information of the container, and fail to fully learn the physical feature values ​​of CT scan images.

[0003] In order to solve such limitations, Patent Document 1 proposes an explosive identification system for realizing rapid and accurate identification of hazardous liquids.

[0004] Patent document 1: CN115452870A Summary of the invention

[0005] However, after research, the inventors of the present application found that the above-mentioned Patent Document 1 still has the following technical problems: it only uses the three-dimensional shape information of the container and the hazardous liquid database to determine whether it is a hazardous liquid, and fails to consider the impact of containers of different materials, different liquid components, etc. on the CT scan image, which can easily cause false detection and missed detection of hazardous liquids in luggage.

[0006] The present application provides a multimodal identification method, device and computer program product for hazardous liquids that considers the influence of container material, liquid composition, etc. in CT scan images, thereby improving the accuracy of hazardous liquid identification.

[0007] One aspect of the present application provides a multimodal identification method for hazardous liquids, the method comprising: generating text data describing the attributes of hazardous liquids for CT imaging image data, the attributes including at least liquid composition and container material; encoding the image data using an image encoder to obtain a visual vector; encoding each attribute in the text data using a text encoder to obtain a text vector, and storing the text vector in a text vector library; and training a multimodal model for identifying hazardous liquids using visual vectors and text vectors for learning image data. By using both visual vectors and text vectors in the training of the multimodal model, and the attributes of the text vectors including at least liquid composition and container material, hazardous liquid identification can be achieved by taking into account the attributes such as liquid composition and container material, thereby improving the accuracy of hazardous liquid identification.

[0008] In some embodiments of the above-mentioned multimodal recognition method, it also includes: for the image data to be tested, according to the trained multimodal model, calculating the similarity between the visual vector and the text vector, and generating a dangerous liquid recognition result containing the descriptive text based on the similarity judgment. In the present application, since the correlation between the visual vector and the text vector is used for recognition, it is possible to recognize the dangerous liquid taking into account the liquid composition, container material and other properties of the dangerous liquid reflected by the text vector, so that the recognition result is more accurate. In addition, since the recognition result contains the descriptive text, and the descriptive text contains keywords of the attributes that match each container in the image data, it is possible to intuitively know the liquid composition, container material and other properties of the dangerous liquid.

[0009] In some embodiments of the above multimodal recognition method, the similarity between the visual vector and the text vector is calculated according to the trained multimodal model, and the hazardous liquid recognition result including the description text is generated according to the similarity judgment, which includes: using the multimodal model, calculating the similarity between the visual vector and the text vector of each attribute in the text vector library; performing the similarity judgment between the visual vector and the text vector; and outputting the recognition result including the description text according to the similarity judgment result. By using the correlation between the visual vector and the text vector for recognition, the accuracy of the hazardous liquid recognition result can be further improved.

[0010] In some embodiments of the above-mentioned multimodal recognition method, the similarity judgment between the visual vector and the text vector includes: setting a preset similarity threshold; and if the similarity is greater than the preset similarity threshold, the text vector of the attribute is recorded as a valid match, and the description text contains keywords corresponding to the valid matching text vector. By setting a threshold and judging the valid matching text vector for each attribute, the description text contains keywords corresponding to the valid matching text vector, and a recognition result with accurate and rich expression content can be generated.

[0011] In some embodiments of the multimodal recognition method described above, encoding the image data using an image encoder to obtain a visual vector includes: obtaining a container area in each image of the image data, and generating a visual vector for each container area. By generating a visual vector for each container area, it is possible to handle the situation where the same image contains multiple containers.

[0012] In some embodiments of the above multimodal recognition method, using a text encoder to encode each attribute in the text data to obtain a text vector includes: segmenting the text data to obtain keywords of different attributes; and encoding the keywords using the text encoder to generate a text vector corresponding to each keyword. In this way, an accurate text vector corresponding to each attribute can be obtained.

[0013] In some embodiments of the above multimodal recognition method, in the text vector library, keywords are grouped according to different attributes, and text vectors belonging to the same attribute are stored as a set, which is beneficial to the management and retrieval of text vectors.

[0014] In some embodiments of the above-mentioned multimodal recognition method, the multimodal model for identifying dangerous liquids using visual vectors and text vectors for training includes: designing a multimodal fusion alignment network to map the visual vectors and text vectors to a common multidimensional feature space; constructing an image-text contrast loss function to optimize the cross-modal alignment of the visual vectors and text vectors, the dimension of the loss function being related to the number of attributes; and training the multimodal fusion alignment network through the loss function to obtain a multimodal model. Since the attributes are also considered in the process of constructing the loss function and the dimension of the loss function is related to the number of attributes, a multimodal model that can consider multiple attributes of the dangerous liquid, such as the liquid composition and container material, can be obtained, thereby making the recognition accuracy using the multimodal model higher.

[0015] In some embodiments of the above-mentioned multimodal recognition method, the multimodal model for identifying hazardous liquids using visual vectors and text vectors for training includes: for each attribute's text vector, the text vector is independently and separately associated with the visual vector. By independently and separately associated learning between the text vector of each attribute and the visual vector, mutual interference between different attribute vectors is avoided, thereby improving the recognition accuracy of the multimodal model.

[0016] In some embodiments of the above multimodal identification method, the attribute further includes: at least one of the liquid volume and the wall thickness of the container. By considering the liquid volume and the wall thickness of the container, the recognition accuracy of the dangerous liquid can be further improved.

[0017] Another aspect of the present application provides a multimodal identification device for hazardous liquids, the device comprising: a data generation unit, for generating text data describing the attributes of hazardous liquids for CT imaging image data, the attributes at least including liquid composition and container material; a visual vector encoding unit, for encoding the image data using an image encoder to obtain a visual vector; a text vector encoding unit, for encoding each attribute in the text data using a text encoder to obtain a text vector, and storing the text vector in a text vector library; and a training unit, for training a multimodal model for identifying hazardous liquids using visual vectors and text vectors for learning image data. By using both visual vectors and text vectors in the training of the multimodal model, and the attributes of the text vectors at least including liquid composition and container material, hazardous liquid identification can be achieved that takes into account the attributes such as liquid composition and container material, thereby improving the accuracy of hazardous liquid identification.

[0018] In some embodiments of the above-mentioned multimodal recognition device, it also includes: a recognition unit, for the image data to be tested, calculates the similarity between the visual vector and the text vector according to the trained multimodal model, and generates a dangerous liquid recognition result including the description text according to the similarity judgment. In the present application, since the correlation between the visual vector and the text vector is used for recognition, it is possible to recognize the dangerous liquid taking into account the liquid composition, container material and other properties of the dangerous liquid reflected by the text vector, so that the recognition result is more accurate. In addition, since the recognition result includes the description text, and the description text includes keywords of the attributes that match each container in the image data, it is possible to intuitively know the liquid composition, container material and other properties of the dangerous liquid.

[0019] In some embodiments of the above-mentioned multimodal recognition device, the recognition unit includes: a similarity calculation module, which uses a multimodal model to calculate the similarity between the visual vector and the text vector of each attribute in the text vector library; a similarity judgment module, which performs similarity judgment between the visual vector and the text vector; and a recognition result output module, which outputs a recognition result including a description text according to the similarity judgment result. By using the correlation between the visual vector and the text vector for recognition, the accuracy of the dangerous liquid recognition result can be further improved.

[0020] In some embodiments of the above-mentioned multimodal recognition device, the similarity judgment module includes: a threshold setting unit, which sets a preset similarity threshold; and an effective match determination unit, which, if the similarity is greater than the preset similarity threshold, records the text vector of the attribute as a valid match, and the description text contains keywords corresponding to the text vector of the effective match. By setting the threshold and determining the text vector of the effective match for each attribute, the description text contains keywords corresponding to the text vector of the effective match, and a recognition result with accurate and rich expression content can be generated.

[0021] In some embodiments of the multimodal recognition device, the visual vector encoding unit obtains the container area in each image of the image data, and generates a visual vector for each container area. By generating a visual vector for each container area, it is possible to handle the situation where the same image contains multiple containers.

[0022] In some embodiments of the above-mentioned multimodal recognition device, the text vector encoding unit includes: a keyword acquisition module, which performs word segmentation on text data to obtain keywords of different attributes; and a text vector generation module, which uses a text encoder to encode keywords and generates a text vector corresponding to each keyword. In this way, an accurate text vector corresponding to each attribute can be obtained.

[0023] In some embodiments of the above-mentioned multimodal recognition device, the text vector encoding unit further includes a text vector library storage module, which groups keywords according to different attributes in the text vector library and stores text vectors belonging to the same attribute as a set, thereby facilitating the management and retrieval of text vectors.

[0024] In some embodiments of the above-mentioned multimodal recognition device, the training unit includes: a multimodal fusion alignment network design module, which designs a multimodal fusion alignment network to map the visual vector and the text vector to a common multidimensional feature space; a loss function construction module, which constructs an image-text contrast loss function to optimize the cross-modal alignment of the visual vector and the text vector, and the dimension of the loss function is related to the number of attributes; and a model training module, which trains the multimodal fusion alignment network through the loss function to obtain a multimodal model. Since the attributes are also considered in the process of constructing the loss function and the dimension of the loss function is related to the number of attributes, a multimodal model that can consider multiple attributes of the dangerous liquid, such as the liquid composition and container material, can be obtained, so that the recognition accuracy using the multimodal model is higher.

[0025] In some embodiments of the above-mentioned multimodal recognition device, the training unit independently performs association learning on the text vector of each attribute and the visual vector. By independently performing association learning on the text vector of each attribute and the visual vector, mutual interference between different attribute vectors is avoided, thereby improving the recognition accuracy of the multimodal model.

[0026] In some embodiments of the above multi-modal recognition device, the attribute further includes: at least one of the liquid volume and the wall thickness of the container. By considering the liquid volume and the wall thickness of the container, the recognition accuracy of the dangerous liquid can be further improved.

[0027] Another aspect of the present application provides a computer program product, including a computer program, which enables a computer to execute the steps in any embodiment of the above-mentioned multimodal recognition method. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart schematically illustrating an embodiment of a multimodal identification method for hazardous liquids.

[0029] Figure 2 Schematic diagram showing encoding of image data to obtain a visual vector.

[0030] Figure 3 is a flowchart schematically showing a process of encoding text data.

[0031] Figure 4 is a schematic diagram showing the encoding process of text data.

[0032] Figure 5 is a flow chart schematically illustrating an example of a process of training a multimodal model to identify hazardous liquids.

[0033] Figure 6 FIG. 4 is a flow chart showing another embodiment of the multimodal identification method for hazardous liquids of the present application.

[0034] Figure 7 is a flow chart schematically illustrating an example of a process of performing hazardous liquid identification using a trained multimodal model.

[0035] Figure 8 FIG. 4 is a schematic diagram showing a process of another embodiment of a multimodal identification method for hazardous liquids.

[0036] Fig. 9 1 is a schematic structural diagram showing an embodiment of a multimodal identification device for hazardous liquids.

[0037] Fig.10 is a schematic structural diagram showing another embodiment of a multimodal identification device for hazardous liquids.

[0038] Fig.11 is a structural diagram showing an example of a text vector encoding unit of a multimodal recognition device.

[0039] Fig.12 1 is a schematic structural diagram showing an example of a training unit of a multimodal recognition device.

[0040] Fig.13 1 is a schematic structural diagram showing an example of a recognition unit of a multimodal recognition device.

[0041] Fig.14 A schematic structural diagram of an electronic device involved in the present application is shown. DETAILED DESCRIPTION

[0042] The exemplary embodiments or examples of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments or examples set forth herein. On the contrary, these embodiments or examples are provided in order to enable a clearer understanding of the present application.

[0043] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments or examples of the present application described herein can be implemented in an order other than that shown or described. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not limited to the steps or units clearly listed, but may include other steps or units that are not clearly listed. The same or similar numbers in the text represent constituent elements with the same or similar functions.

[0044] In the prior art, since the influence of containers of different materials and different liquid components on CT scan images is not considered, it is easy to cause misdetection and missed detection of dangerous liquids in luggage, thereby affecting the accuracy of dangerous liquid identification.

[0045] The present application provides a multimodal identification method, device and computer program product for hazardous liquids that take into account the influence of container material, liquid composition, etc. in CT scan images, thereby improving the accuracy of hazardous liquid identification.

[0046] One embodiment of the present application provides a multimodal identification method for hazardous liquids.

[0047] Figure 1 is a flow chart schematically illustrating an embodiment of a multimodal identification method for hazardous liquids.

[0048] The following, combined Figure 1 , describing in detail the process of the multimodal identification method for hazardous liquids of the present application.

[0049] In step S10, text data describing the properties of the dangerous liquid is generated for the CT imaging image data, wherein the properties at least include the liquid composition and the container material.

[0050] The reason why the above-mentioned liquid composition and container material are included as attributes is that the inventor of the present application has found through research and experiments that the liquid composition and container material have the most obvious influence on the image recognition of CT scan images. Among them, the liquid composition can also be reflected in atomic number, density, etc.

[0051] Specifically, regarding the container material, each type of liquid in a container will show different characteristics in the CT image data, and there is a unique relationship between the material and the characteristics of the CT image data. For example, gasoline in a glass container and gasoline in a plastic container have different characteristic values ​​in the CT scanned image data. The same is true for liquid composition. Each type of liquid composition will show different characteristics in the CT image data, and there is a unique relationship between the liquid composition and the characteristics of the CT image data.

[0052] This application utilizes this principle to design a multimodal recognition method and device that takes into account liquid composition and container material.

[0053] The inventor of the present application has found through further research that other attributes may also affect the image recognition result. Therefore, in addition to the above-mentioned liquid composition and container material, at least one of the liquid volume and container wall thickness may also be included as an attribute.

[0054] Different attributes can also be called different dimensions.

[0055] For example, when the attributes include liquid composition, container material, and liquid volume, the text data may be “300 ml of plastic bottled gasoline”.

[0056] In some embodiments, three-dimensional image data of CT scans related to liquid composition and container material can be obtained by uniform sampling, and then the three-dimensional CT image data can be cropped to obtain each image data; for each image data, text data describing the properties of the dangerous liquid in the image can be generated. For example, the above-mentioned multiple properties can be annotated by multimodal annotation, and text data can be generated based on the annotated text.

[0057] In some embodiments, each image may also be paired with text data to form an image-text pair dataset.

[0058] In step S20, an image encoder is used to encode the image data to obtain a visual vector.

[0059] In some embodiments, an image encoder can be designed to extract the core visual structure from the CT image, and the image encoder can include a three-dimensional visual detection network to achieve accurate recognition of different container geometric features. In addition, an existing image encoder can also be used, and this application does not limit this.

[0060] In some embodiments, Figure 2As shown, a three-dimensional visual detection network is used to process CT image data, obtain the container area in each image of the image data, and generate a visual vector for each container area. By generating a visual vector for each container area, it is possible to handle the situation where multiple containers are contained in the same image.

[0061] Optionally, each container region can be converted into a high-dimensional visual vector. By encoding image data into a high-dimensional visual vector, richer information can be included, which is conducive to the correlation learning between the visual vector and the text vector.

[0062] In step S30, a text encoder is used to encode each attribute in the text data to obtain a text vector, and the text vector is stored in a text vector library.

[0063] In some embodiments, a text encoder may be designed to understand and encode text information. The text encoder may use a natural language model of a text transformer architecture. In addition, existing text encoders may also be used, and other natural language models may also be used in these text editors, which is not limited in this application.

[0064] Figure 3 is a flow chart schematically showing a text data encoding process.

[0065] like Figure 3 As shown, step S30 may specifically include: step S31, step S32, and step S33.

[0066] In step S31, the text data is segmented to obtain keywords of different attributes.

[0067] In some embodiments, text data in the image-text pair dataset may be segmented to obtain keywords of different dimensions.

[0068] For example, “500ml plastic bottled gasoline” can be segmented into keywords [gasoline, plastic, 500ml, …].

[0069] For example, different attributes (dimensions) and keywords corresponding to each attribute may include: [gasoline, plastic, 500ml, ...], [sulfuric acid, glass, 300ml, ...], [water, plastic, 1L, ...].

[0070] In this case, the following keywords can be segmented:

[0071] The keywords for the liquid composition dimension are: [gasoline], [sulfuric acid], [water];

[0072] The keywords for the container material dimension are: [plastic], [glass], [plastic];

[0073] The keywords for the liquid volume dimension are: [500ml], [300ml], [1L].

[0074] By segmenting the text data, we obtained keywords for different attributes such as liquid composition, container material, and liquid volume.

[0075] In step S32, a text encoder is used to encode the keywords, and a text vector is generated corresponding to each keyword.

[0076] As an application example of the above-mentioned step S31 and step S32, Figure 4 As shown, "300 ml of gasoline in plastic" can be segmented to obtain keywords such as "gasoline", "plastic", and "300 ml", and the text encoder can be used to encode the above keywords such as "gasoline", "plastic", and "300 ml", and a text vector is generated for each keyword.

[0077] By segmenting the text data to obtain keywords of different attributes, encoding the keywords and correspondingly generating a text vector, an accurate text vector corresponding to each attribute can be obtained.

[0078] In step S33, the text vector is stored in a text vector library.

[0079] In the text vector library, keywords can be grouped according to different attributes, and text vectors belonging to the same attributes can be stored as a set.

[0080] For example, keywords indicating liquid components such as "gasoline", "sulfuric acid", and "water" are grouped together and stored as a set.

[0081] As an example, a collection of text vectors describing the composition of liquids:

[0082] Embed liquid ingredients [gasoline, kerosene, sulfuric acid, hydrochloric acid, water, beverages, ...]

[0083] For example, keywords indicating container materials such as "glass", "plastic", and "ceramic" are grouped together and stored as a set.

[0084] As an example, a collection of text vectors describing the container material:

[0085] Embed 容器材质 [Glass, plastic, ceramic, metal, rubber, wood, ...]

[0086] In addition to the above-mentioned liquid composition and container material, keywords representing liquid volume may also be grouped together and stored as a set.

[0087] As an example, a collection of text vectors describing the volume of liquids:

[0088] Embed 液体体积 [100ml, 200ml, 500ml, 1L, 2L, 3L,…]

[0089] The text vector library may be pre-built or built when training the multimodal model, and this application does not limit this. In addition, when training the multimodal model and recognizing the image data to be tested, the text vector library may also be updated, so that the text vector library can recognize texts with more attributes through continuous learning.

[0090] By grouping keywords according to different attributes and storing text vectors belonging to the same attribute as a set, it is beneficial to the management and retrieval of text vectors.

[0091] Next, return Figure 1 The description will be continued. In step S40, a multimodal model for identifying dangerous liquids is trained using visual vectors and text vectors for image data used for learning.

[0092] Figure 5 is a flow chart schematically illustrating an example of a process of training a multimodal model to identify hazardous liquids.

[0093] like Figure 5 As shown, the specific process of training a multimodal model may include:

[0094] In step S41, a multimodal fusion alignment network is designed to map the visual vector and text vector into a common multidimensional feature space.

[0095] In some embodiments, by designing a multimodal fusion alignment network, the visual vector generated by the image encoder and the text vector generated by the text encoder can be mapped to a common multidimensional feature space, thereby achieving matching and correspondence between the visual vector and the text vector.

[0096] In step S42, an image-text contrast loss function is constructed to optimize the cross-modal alignment of visual vectors and text vectors.

[0097] By constructing an image-text contrast loss function and further optimizing the cross-modal alignment of visual vectors and text vectors, image-text pairing learning can be effectively achieved.

[0098] The dimension of the loss function is related to the number of attributes. That is, when three attributes are used, a three-dimensional loss function can be constructed, and the weight parameters of the three dimensions are determined through continuous learning.

[0099] By considering the number of attributes in the construction of the loss function, the relationship between each attribute in the text vector and the visual vector can be reflected, and each attribute can be comprehensively considered to obtain the recognition result, thereby improving the accuracy of the hazardous liquid recognition results.

[0100] In step S43, the multimodal fusion alignment network is trained through the loss function to obtain a multimodal model.

[0101] In this application, a multimodal model is trained by designing a multimodal fusion alignment network, constructing an image-text contrast loss function, and training the multimodal fusion alignment network through the loss function, thereby obtaining a model in which the visual vector and the text vector are matched.

[0102] In addition, since its text vector takes into account multiple attributes of hazardous liquids, such as the liquid composition and container material, and these attributes are also considered in the construction of the loss function, a multimodal model that can consider multiple attributes of hazardous liquids, such as the liquid composition and container material, can be obtained, thereby making the recognition accuracy using the multimodal model higher.

[0103] In addition, in some embodiments, during the process of training the multimodal model, the text vector of each attribute is independently learned to be associated with the visual vector.

[0104] Specifically, the text attributes of each dimension are learned to correlate with the image features, and the correlation models learned in different dimensions are independent of each other. When identifying dangerous liquids, by describing their characteristics in different dimensions, and without interference between different dimensions, a more accurate judgment can be made by comprehensively considering multiple dimensions.

[0105] After the multimodal recognition model is trained, the trained multimodal recognition model can also be used to identify hazardous liquids.

[0106] In this embodiment, by using both visual vectors and text vectors in the training of the multimodal model, wherein the attributes of the text vector include at least liquid composition and container material, it is possible to realize the identification of hazardous liquids taking into account the attributes such as liquid composition and container material, thereby improving the accuracy of hazardous liquid identification.

[0107] Figure 6 FIG. 1 is a flow chart showing another embodiment of the multimodal identification method for hazardous liquids of the present application. Figure 6 In addition to Figure 1 In addition to steps S10 to S40, the method also includes step S50 for identifying hazardous liquids using the trained multimodal model.

[0108] In step S50, for the image data to be tested, the similarity between the visual vector and the text vector is calculated according to the trained multimodal model, and based on the similarity judgment, a hazardous liquid recognition result including a description text is generated.

[0109] Figure 7 is a flow chart schematically illustrating an example of a process of performing hazardous liquid identification using a trained multimodal model.

[0110] like Figure 7 As shown, the specific process of using the trained multimodal model to identify hazardous liquids may include steps S51 to S53.

[0111] In step S51, the similarity between the visual vector and the text vector of each attribute in the text vector library is calculated.

[0112] In some embodiments, Figure 7 As shown, the visual vector and the text vector can be input into the trained multimodal model, and the multimodal model is used to calculate the similarity between the visual vector and the text vector of each attribute in the text vector library.

[0113] In step S52, the similarity between the visual vector and the text vector is judged.

[0114] In some embodiments, similarity determination may be performed on the visual vector of each container region of each image and on the text vector of each keyword.

[0115] In some embodiments, the specific process of similarity determination may include:

[0116] Set a preset similarity threshold;

[0117] When the similarity calculated in step S52 is greater than a preset similarity threshold, the corresponding keyword, ie, the corresponding text vector, is recorded as a valid match of the visual vector.

[0118] In each text vector set for the same attribute, at most one valid match is retained. That is, the valid text vector in each attribute can only be unique.

[0119] If there is no valid match in the text vector set of a certain attribute, the keywords of the attribute corresponding to the set can be omitted when the description text is subsequently generated.

[0120] In step S53, based on the similarity judgment result, the recognition result including the description text is output.

[0121] In some embodiments, for each attribute set, keywords related to the valid matching text vectors may be included in the description text. The description text may be generated by searching for keywords related to the valid matching text vectors for multiple attribute sets respectively and combining these keywords.

[0122] The identification result of the hazardous liquid usually also includes the spatial position of the hazardous liquid. Therefore, it can be understood that the above identification result includes the spatial position of the hazardous liquid and the description text related to the hazardous liquid.

[0123] By setting a threshold and determining the valid matching text vector for each attribute, the description text contains keywords corresponding to the valid matching text vector, and a recognition result with accurate and rich expression content can be generated.

[0124] Figure 8 A schematic diagram showing another embodiment of a multimodal identification method for hazardous liquids.

[0125] Figure 8 The whole process is shown, from generating text data, performing text vector encoding (text encoding) and visual vector encoding (image encoding) based on the text data and CT image data to generate text vectors and visual vectors, and inputting the text vectors and visual vectors into the trained multimodal recognition model to obtain the hazardous liquid recognition results.

[0126] In the present application, since the correlation between the visual vector and the text vector is used for identification, the dangerous liquid can be identified taking into account the liquid composition, container material and other properties of the dangerous liquid reflected by the text vector, so that the identification result is more accurate. In addition, since the recognition result includes the description text, and the description text includes the keywords of the properties that match each container in the image data, the liquid composition, container material and other properties of the dangerous liquid can be intuitively known.

[0127] In addition, the above identification results can also be sent to a baggage inspection system, etc. for subsequent comprehensive inspection.

[0128] Another embodiment of the present application provides a multimodal identification device for hazardous liquids.

[0129] Fig. 9 1 is a schematic structural diagram showing an embodiment of a multimodal identification device for hazardous liquids.

[0130] like Fig. 9 As shown, the multimodal identification device for hazardous liquids may include: a data generating unit 10 , a visual vector encoding unit 20 , a text vector encoding unit 30 , and a training unit 40 .

[0131] The data generating unit 10 generates text data describing the properties of the dangerous liquid based on the image data of the CT imaging, where the properties at least include liquid composition and container material.

[0132] The visual vector encoding unit 20 encodes the image data using an image encoder to obtain a visual vector.

[0133] The visual vector encoding unit 20 may obtain a container region in each image of the image data, and generate a visual vector for each container region.

[0134] The text vector encoding unit 30 uses a text encoder to encode each attribute in the text data to obtain a text vector, and stores the text vector in a text vector library.

[0135] The training unit 40 trains a multimodal model for identifying dangerous liquids using the visual vector and the text vector for the learning image data.

[0136] Fig.10 is a schematic structural diagram showing another embodiment of a multimodal identification device for hazardous liquids. Fig.10 The multimodal identification device for hazardous liquids shown in the figure has Fig. 9 In addition to the data generating unit 10 , the visual vector encoding unit 20 , the text vector encoding unit 30 , and the training unit 40 , the illustrated system further includes a recognition unit 50 .

[0137] The recognition unit 50 calculates the similarity between the visual vector and the text vector for the image data to be tested according to the trained multimodal model, and generates a dangerous liquid recognition result including a description text according to the similarity judgment.

[0138] The above-mentioned data generation unit 10, visual vector encoding unit 20, text vector encoding unit 30, training unit 40, and recognition unit 50 respectively perform the processing in steps S10 to S50 in the above-mentioned multimodal recognition method for hazardous liquids. Therefore, its details and technical effects can refer to the above-mentioned implementation method of the multimodal recognition method.

[0139] Fig.11 2 is a schematic structural diagram showing an example of the text vector encoding unit 30 .

[0140] like Fig.11 As shown, the text vector encoding unit 30 may include: a keyword acquisition module 31 , a text vector generation module 32 , and a text vector library storage module 33 .

[0141] The keyword acquisition module 31 performs word segmentation on the text data to acquire keywords of different attributes.

[0142] The text vector generation module 32 uses the text encoder to encode the keywords and generates a text vector corresponding to each keyword.

[0143] The text vector library storage module 33 groups keywords according to different attributes in the text vector library, and stores text vectors belonging to the same attribute as a set.

[0144] The above-mentioned keyword acquisition module 31, text vector generation module 32, and text vector library storage module 33 respectively perform the processing in steps S31 to S33 in the above-mentioned multimodal identification method for hazardous liquids. Therefore, their details and technical effects can refer to the above-mentioned implementation method of the multimodal identification method.

[0145] Fig.12 2 is a schematic structural diagram showing an example of the training unit 40 .

[0146] like Fig.12 As shown, the training unit 40 may include: a multimodal fusion alignment network design module 41, a loss function construction module 42, and a model training module 43.

[0147] The multimodal fusion alignment network design module 41 designs a multimodal fusion alignment network to map the visual vector and the text vector to a common multidimensional feature space;

[0148] The loss function construction module 42 constructs an image-text contrast loss function to optimize the cross-modal alignment of the visual vector and the text vector, wherein the dimension of the loss function is related to the number of attributes;

[0149] The model training module 43 trains the multimodal fusion alignment network through a loss function to obtain a multimodal model.

[0150] The training unit 40 also independently performs association learning between the text vector of each attribute and the visual vector.

[0151] The above-mentioned multimodal fusion alignment network design module 41, loss function construction module 42, and model training module 43 respectively perform the processing in steps S41 to S43 in the above-mentioned multimodal identification method for hazardous liquids. Therefore, their details and technical effects can refer to the above-mentioned implementation method of the multimodal identification method.

[0152] Fig.13 2 is a schematic structural diagram showing an example of the recognition unit 50 .

[0153] like Fig.13 As shown, the recognition unit 50 may further include: a similarity calculation module 51 , a similarity judgment module 52 , and a recognition result output module 53 .

[0154] The similarity calculation module 51 calculates the similarity between the visual vector and the text vector of each attribute in the text vector library.

[0155] The similarity judgment module 52 performs similarity judgment on the visual vector and the text vector.

[0156] The recognition result output module 53 outputs the recognition result including the description text according to the similarity judgment result.

[0157] The similarity judgment module 51 may further include:

[0158] A threshold setting unit, which sets a preset similarity threshold; and

[0159] The effective match determination unit records the text vector of the attribute as an effective match if the similarity is greater than a preset similarity threshold.

[0160] The description text contains keywords corresponding to the effectively matched text vectors.

[0161] The above-mentioned similarity calculation module 51, similarity judgment module 52, and recognition result output module 53 respectively perform the processing in steps S51 to S53 in the above-mentioned multimodal recognition method for hazardous liquids. Therefore, their details and technical effects can refer to the above-mentioned implementation method of the multimodal recognition method.

[0162] One embodiment of the present application may also provide an electronic device.

[0163] Fig.14 FIG. 1 is a schematic diagram showing the structure of an electronic device involved in the present application. Fig.14 As shown, the electronic device may include a processor 801 and a memory 802 storing computer programs or instructions.

[0164] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0165] The memory 802 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 802 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 802 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 802 is a non-volatile solid-state memory. In a specific embodiment, the memory 802 includes a read-only memory (ROM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0166] The processor 801 reads and executes computer program instructions stored in the memory 802 to implement any one of the multimodal identification methods for hazardous liquids in the above embodiments.

[0167] In one example, the electronic device may further include a communication interface 803 and a bus 810. Fig.14 As shown, the processor 801, the memory 802, and the communication interface 803 are connected via a bus 810 and communicate with each other.

[0168] The communication interface 803 is mainly used to implement communication between various modules, devices, units and / or devices in the embodiments of the present application.

[0169] Bus 810 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 810 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0170] The electronic device can execute the multimodal identification method of the hazardous liquid of the present application, thereby realizing the multimodal identification device of the hazardous liquid of the present application.

[0171] In addition, in combination with the above multi-modal identification method for hazardous liquids, the present application may also provide a readable storage medium for implementation. The readable storage medium stores program instructions; when the program instructions are executed by a processor, any of the multi-modal identification methods for hazardous liquids in the above embodiments is implemented.

[0172] In combination with the above multi-modal identification method for hazardous liquids, the present application may also provide a computer program product for implementation. The computer program product includes a computer program, and when the program instructions are executed by a processor, any multi-modal identification method for hazardous liquids in the above embodiments is implemented.

[0173] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0174] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0175] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or devices based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0176] The above description is only a specific implementation mode of the present application. Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] It should be understood that the protection scope of the present application is not limited thereto, and any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and these modifications or substitutions should be included in the protection scope of the present application.

Claims

1. A multimodal identification method for hazardous liquids, comprising: Generating text data describing the properties of the dangerous liquid for the CT imaging image data, wherein the properties at least include the liquid composition and the container material; Using an image encoder, encoding the image data to obtain a visual vector; Using a text encoder, encoding each attribute in the text data to obtain a text vector, and storing the text vector in a text vector library; and For the image data used for learning, the visual vector and the text vector are used to train a multimodal model for identifying dangerous liquids.

2. The multimodal recognition method according to claim 1, further comprising: For the image data to be tested, the similarity between the visual vector and the text vector is calculated according to the trained multimodal model, and a dangerous liquid recognition result containing a description text is generated based on the similarity judgment.

3. The multimodal recognition method according to claim 2, wherein: Calculating the similarity between the visual vector and the text vector according to the trained multimodal model, and generating a dangerous liquid recognition result including a description text according to the similarity judgment includes: Utilizing the multimodal model, calculating the similarity between the visual vector and the text vector of each attribute in the text vector library; Performing similarity judgment between the visual vector and the text vector; and Based on the similarity judgment results, the recognition results containing the description text are output.

4. The multimodal recognition method according to claim 3, wherein: The similarity determination between the visual vector and the text vector includes: Setting a preset similarity threshold; and If the similarity is greater than a preset similarity threshold, the text vector of the attribute is recorded as a valid match. The description text contains keywords corresponding to the effectively matched text vectors.

5. The multimodal recognition method according to claim 1 or 2, wherein: Using an image encoder, encoding the image data to obtain a visual vector includes: The container area in each image in the image data is obtained, and the visual vector is generated for each container area.

6. The multimodal recognition method according to claim 1 or 2, wherein: Using a text encoder, encoding each attribute in the text data to obtain a text vector includes: Segmenting the text data to obtain keywords of different attributes; and The keywords are encoded using the text encoder, and a text vector is generated corresponding to each keyword.

7. The multimodal recognition method according to claim 6, wherein: In the text vector library, keywords are grouped according to different attributes, and text vectors belonging to the same attribute are stored as a set.

8. The multimodal recognition method according to claim 1, wherein: Using the visual vector and the text vector to train a multimodal model for identifying dangerous liquids includes: Designing a multimodal fusion alignment network to map the visual vector and the text vector into a common multidimensional feature space; Constructing an image-text contrast loss function to optimize the cross-modal alignment of the visual vector and the text vector, wherein the dimension of the loss function is related to the number of attributes; and The multimodal fusion alignment network is trained through the loss function to obtain a multimodal model.

9. The multimodal recognition method according to claim 1 or 8, wherein: Using the visual vector and the text vector to train a multimodal model for identifying dangerous liquids includes: For each attribute's text vector, correlation learning is performed independently with the visual vector.

10. The multimodal recognition method according to claim 1, wherein: The properties also include at least one of: liquid volume, and wall thickness of the container.

11. A multimodal identification device for hazardous liquids, comprising: A data generating unit, for generating text data describing properties of the dangerous liquid based on the CT imaging image data, wherein the properties at least include liquid composition and container material; A visual vector encoding unit, using an image encoder to encode the image data to obtain a visual vector; A text vector encoding unit, using a text encoder, encodes each attribute in the text data to obtain a text vector, and stores the text vector in a text vector library; and The training unit trains a multimodal model for identifying dangerous liquids using the visual vector and the text vector for the image data used for learning.

12. The multimodal recognition device according to claim 11, further comprising: The recognition unit calculates the similarity between the visual vector and the text vector according to the trained multimodal model for the image data to be tested, and generates a dangerous liquid recognition result containing a description text according to the similarity judgment.

13. The multimodal recognition device according to claim 12, wherein: The identification unit comprises: A similarity calculation module, using the multimodal model, calculates the similarity between the visual vector and the text vector of each attribute in the text vector library; a similarity judgment module, performing similarity judgment between the visual vector and the text vector; and The recognition result output module outputs the recognition result including the description text according to the similarity judgment result.

14. The multimodal recognition device according to claim 13, wherein: The similarity judgment module includes: A threshold setting unit, which sets a preset similarity threshold; and The effective match determination unit records the text vector of the attribute as an effective match if the similarity is greater than a preset similarity threshold. The description text contains keywords corresponding to the effectively matched text vectors.

15. The multimodal recognition device according to claim 11 or 12, wherein: The visual vector encoding unit acquires a container area in each image of the image data, and generates the visual vector for each container area.

16. The multimodal recognition device according to claim 11 or 12, wherein: The text vector encoding unit comprises: A keyword acquisition module, which performs word segmentation on the text data to acquire keywords of different attributes; and The text vector generation module uses the text encoder to encode the keywords and generates a text vector corresponding to each keyword.

17. The multimodal recognition device according to claim 16, wherein: The text vector encoding unit also includes a text vector library storage module, The text vector library storage module groups keywords according to different attributes in the text vector library, and stores text vectors belonging to the same attribute as a set.

18. The multimodal recognition device according to claim 11, wherein: The training section includes: A multimodal fusion alignment network design module designs a multimodal fusion alignment network to map the visual vector and the text vector to a common multidimensional feature space; A loss function construction module is used to construct an image-text contrast loss function to optimize the cross-modal alignment of the visual vector and the text vector, wherein the dimension of the loss function is related to the number of attributes; and The model training module trains the multimodal fusion alignment network through the loss function to obtain a multimodal model.

19. The multimodal recognition device according to claim 11 or 18, wherein: The training unit independently performs association learning on the text vector of each attribute and the visual vector.

20. The multimodal recognition device according to claim 11, wherein: The properties also include at least one of: liquid volume, and wall thickness of the container.

21. A computer program product, comprising a computer program, wherein the program enables a computer to execute the steps of the multimodal identification method for hazardous liquids as claimed in claims 1 to 10.

Images